[Question] Can I build AWS SDK CPP with static libcurl?

[sunhyung]

What platform/OS are you using?

Ubuntu 18.04 LTS

Which version of the SDK?

1.7.162

What compiler are you using? what version?

CLang 6

What are your CMake arguments?

cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE="RelWithDebInfo" -DCMAKE_CXX_COMPILER:FILEPATH="clang++" -DCMAKE_C_COMPILER:FILEPATH="clang" -DENABLE_TESTING="OFF" -DBUILD_ONLY=${AWS_FEATURES} -DBUILD_SHARED_LIBS="OFF"

Can you provide a TRACE level log? (sanitize any sensitive information)

[sunhyung]

Due to curl 7.58 issue I posted https://github.com/aws/aws-sdk-cpp/issues/1253, I'm trying to build my project with the latest curl which is 7.66. I successfully built curl 7.66 locally and made simple project that links static curl library which is libcurl.a. (Curl is installed in /home/me/curl) But I have no idea how to build AWS SDK CPP with static curl library.

I've built AWS SDK with additional parameters which are -DCURL_LIBRARY="/home/me/curl/lib/libcurl.a" -DCURL_INCLUDE_DIR="/home/me/curl/include" but my application is still linked with libcurl.so.

Is there something build option for AWS SDK CPP to use static curl library in Linux?

[KaibaLopez]

Hi @sunhyung , You should be able to, try adding the cmake argument -DCMAKE_PREFIX_PATH="home/me/curl" That should take the static lib that curl was built against. Let me know if that works for you, but if it doesn't, please provide the error log.

[sunhyung]

Hi @sunhyung , You should be able to, try adding the cmake argument -DCMAKE_PREFIX_PATH="home/me/curl" That should take the static lib that curl was built against. Let me know if that works for you, but if it doesn't, please provide the error log.

Thanks for the quick response!! I will try it!

[sunhyung]

I've cleaned my WSL Ubuntu and built curl 7.66 and AWS SDK CPP again. I confirmed AWS SDK built well with custom built curl 7.66 but the test project still links with dynamic libcurl.so Here's CMakeLists.txt for the test project. If there are something to be added, please let me know.

cmake_minimum_required (VERSION 3.8)

set(CMAKE_VERBOSE_MAKEFILE ON)

find_package(AWSSDK REQUIRED COMPONENTS core)

file(GLOB SRC ".h" ".cpp" )

add_executable(awssdk_test ${SRC})

target_link_libraries(awssdk_test ${AWSSDK_LINK_LIBRARIES} )

[sunhyung]

These are ldd result.

linux-vdso.so.1 (0x00007ffff9c9a000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f469c720000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f469c250000) libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f469bfc0000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f469bda0000) libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f469bb20000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f469b780000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f469b560000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f469b1d0000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f469afb0000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f469abb0000) /lib64/ld-linux-x86-64.so.2 (0x00007f469ca00000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f469a9a0000) libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f469a770000) libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f469a540000) librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f469a320000) libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f469a110000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f4699ec0000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f4699c60000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f4699a50000) libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f46996c0000) libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f4699350000) libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f4699110000) libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f4698ed0000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f4698c40000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f4698960000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f4698710000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f4698500000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f46982f0000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f46980d0000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f4697eb0000) libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f4697c60000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f4697920000) libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f4697700000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f46974f0000) libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f46972e0000) libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f4697050000) libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f4696d90000) libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f4696b50000) libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f4696930000) libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f4696720000) libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f46964f0000) libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f46962e0000) libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f4696080000) libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f4695d70000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f4695b30000)

[sunhyung]

Also I printed ${AWSSDK_LINK_LIBRARIES} right after find_package and it tells like this,

CMake codes find_package(AWSSDK REQUIRED COMPONENTS core) message("================ " ${AWSSDK_LINK_LIBRARIES})

Result ================ aws-cpp-sdk-corepthreadcryptosslzcurl

Is it okay? I don't see any white spaces between libraries. This result shows curl as a dynamic library, not my local built custom libcurl.a

[sunhyung]

I think I kind of figured out how to link libcurl.a. After installing AWS SDK CPP, I edited AWSSDK_CLIENT_LIBS value in platformDeps.cmake file which was "curl" but I changed it with my latest custom built libcurl.a path then the test application links static curl library. Is there a way to set AWSSDK_CLIENT_LIBS when I build AWS SDK?

[sunhyung]

Well... I don't think there is a way to change AWSSDK_CLIENT_LIBS value of platformDeps.cmake due to line 97 of https://github.com/aws/aws-sdk-cpp/blob/master/cmake/external_dependencies.cmake file.

[singku]

When you run cmake scripts of cpp sdk, the output should tell you the found curl. Is it stil finding the old system one?

[sunhyung]

When you run cmake scripts of cpp sdk, the output should tell you the found curl. Is it stil finding the old system one?

When I build AWS CPP SDK, the output tells my local build libcurl.a. These are a part of sdk build output after sanitizing.

-- The CXX compiler identification is Clang 6.0.0 -- Check for working CXX compiler: /usr/bin/clang++ -- Check for working CXX compiler: /usr/bin/clang++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") -- Zlib include directory: /usr/include -- Zlib library: /usr/lib/x86_64-linux-gnu/libz.so -- Encryption: Openssl -- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1") -- Openssl include directory: /usr/include -- Openssl library: /usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so -- Http client: Curl -- Found CURL: /mnt/d/myproject/3rdParty/curl/lib/libcurl.a (found version "7.66.0") -- Curl include directory: /mnt/d/myproject/3rdParty/curl/include -- Curl library: /mnt/d/myproject/3rdParty/curl/lib/libcurl.a -- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC -- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC - Success -- Considering dynamodb

[sunhyung]

This is how I build curl 7.66 locally,

cd ~ wget http://curl.haxx.se/download/curl-7.66.0.tar.gz && tar -xvf curl-7.66.0.tar.gz mkdir curl-build && cd curl-build cmake ../curl-7.66.0 -DCMAKE_BUILD_TYPE="RelWithDebInfo" -DCMAKE_CXX_COMPILER:FILEPATH="clang++" -DCMAKE_C_COMPILER:FILEPATH="clang" -DCMAKE_INSTALL_PREFIX="../curl" -DBUILD_SHARED_LIBS="OFF" -DBUILD_CURL_EXE="OFF" make && make install

After all these commands, it would make ~/curl path and put every built library & include files in it then I copied ~/curl and its files to my 3rdParty/curl under my project.

[sunhyung]

This is full log of building cpp sdk aws-sdk-build.log

[sunhyung]

This is my shell script to build cpp sdk.

!/bin/bash

AWS_FEATURES="dynamodb;firehose;gamelift" CURL_PATH=$(pwd)/curl

cd aws-sdk-build-linux

sudo cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE="RelWithDebInfo" -DCMAKE_CXX_COMPILER:FILEPATH="clang++" -DCMAKE_C_COMPILER:FILEPATH="clang" -DENABLE_TESTING="OFF" -DBUILD_ONLY=${AWS_FEATURES} -DBUILD_SHARED_LIBS="OFF" -DCMAKE_PREFIX_PATH=$CURL_PATH

sudo cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE="RelWithDebInfo" -DCMAKE_CXX_COMPILER:FILEPATH="clang++" -DCMAKE_C_COMPILER:FILEPATH="clang" -DENABLE_TESTING="OFF" -DBUILD_ONLY=${AWS_FEATURES} -DBUILD_SHARED_LIBS="OFF" -DCMAKE_INSTALL_PREFIX="../aws-sdk-cpp/install/Linux/RelWithDebInfo" -DCMAKE_PREFIX_PATH=$CURL_PATH

sudo make && sudo make install cd .. rm -rf aws-sdk-build-linux

[KaibaLopez]

Hi @sunhyung, Can you try modifying curl configure in the curl-7.66.0 folder? LIBS=-lpthread ./configure --disable-shared and then try to build the sdk, let me know if that works?

[sunhyung]

FYI, my Ubuntu 18.04 already has Curl 7.58 in common path.

dpkg -L libcurl4-openssl-dev /. /usr /usr/bin /usr/bin/curl-config /usr/include /usr/include/x86_64-linux-gnu /usr/include/x86_64-linux-gnu/curl /usr/include/x86_64-linux-gnu/curl/curl.h /usr/include/x86_64-linux-gnu/curl/curlver.h /usr/include/x86_64-linux-gnu/curl/easy.h /usr/include/x86_64-linux-gnu/curl/mprintf.h /usr/include/x86_64-linux-gnu/curl/multi.h /usr/include/x86_64-linux-gnu/curl/stdcheaders.h /usr/include/x86_64-linux-gnu/curl/system.h /usr/include/x86_64-linux-gnu/curl/typecheck-gcc.h /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/libcurl.a /usr/lib/x86_64-linux-gnu/libcurl.la /usr/lib/x86_64-linux-gnu/pkgconfig /usr/lib/x86_64-linux-gnu/pkgconfig/libcurl.pc /usr/share /usr/share/aclocal /usr/share/aclocal/libcurl.m4 /usr/share/doc /usr/share/doc/libcurl4-openssl-dev /usr/share/doc/libcurl4-openssl-dev/copyright /usr/share/man /usr/share/man/man1 /usr/share/man/man1/curl-config.1.gz /usr/lib/x86_64-linux-gnu/libcurl.so /usr/share/doc/libcurl4-openssl-dev/NEWS.Debian.gz /usr/share/doc/libcurl4-openssl-dev/changelog.Debian.gz

[KaibaLopez]

Hi @sunhyung sorry but, did you try what I suggested with the disable shared library configuration for curl?

[sunhyung]

Hi @sunhyung sorry but, did you try what I suggested with the disable shared library configuration for curl?

Yeah, I successfully built CPP SDK but I haven't checked the build log yet. As soon as I check, I will put the result here. :)

[sunhyung]

Hi @sunhyung sorry but, did you try what I suggested with the disable shared library configuration for curl?

It's just same as I did yesterday. FYI, I built AWS SDK CPP as static library.

[sunhyung]

I found the code below in setup_cmake_find_module.cmake and it tells "set(AWSSDK_CLIENT_LIBS ${CLIENT_LIBS_ABSTRACT_NAME})\n". As long as this line exists, I think AWS CPP SDK keeps looking for dynamic curl. file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}/platformDeps.cmake" "# # Copyright 2010-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the \"License\"). # You may not use this file except in compliance with the License. # A copy of the License is located at # # http://aws.amazon.com/apache2.0 # # or in the \"license\" file accompanying this file. This file is distributed # on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either # express or implied. See the License for the specific language governing # permissions and limitations under the License. #\n" "set(AWSSDK_PLATFORM_DEPS_LIBS ${PLATFORM_DEP_LIBS_ABSTRACT_NAME})\n" "set(AWSSDK_CLIENT_LIBS ${CLIENT_LIBS_ABSTRACT_NAME})\n" "set(AWSSDK_CRYPTO_LIBS ${CRYPTO_LIBS_ABSTRACT_NAME})\n" "set(AWSSDK_ADDITIONAL_LIBS ${AWS_SDK_ADDITIONAL_LIBRARIES_ABSTRACT_NAME})\n" "set(AWSSDK_INSTALL_LIBDIR ${LIBRARY_DIRECTORY})\n" "set(AWSSDK_INSTALL_BINDIR ${BINARY_DIRECTORY})\n" "set(AWSSDK_INSTALL_INCLUDEDIR ${INCLUDE_DIRECTORY})\n" "set(AWSSDK_INSTALL_ARCHIVEDIR ${ARCHIVE_DIRECTORY})\n" )

[singku]

It tells you which libs (names) sdk depends on, not the absolute paths of the exact libs. The reason we do this is that you could later update your system and make the absolute paths unreliable. In your case, you just need to build static libs only then build sdk with the static libs, and when linking your own application, tell cmake where to find static libs as well.

[sunhyung]

It tells you which libs (names) sdk depends on, not the absolute paths of the exact libs. The reason we do this is that you could later update your system and make the absolute paths unreliable. In your case, you just need to build static libs only then build sdk with the static libs, and when linking your own application, tell cmake where to find static libs as well.

Does it mean that I should link AWS SDK and "locally built curl" in my application? If so... it will be so sad...

[KaibaLopez]

Sadly yes. It's really more of a cmake thing than the sdk's I think... kind of adding to @singku 's explanation but, you have to tell cmake that you're using the static libraries, I think you could do something like :

add_executable(awssdk_test ${SRC})
target_link_directories(awssdk_test /path/to/customized/static/curl/lib BEFORE) #the important part.
target_link_libraries(awssdk_test
${AWSSDK_LINK_LIBRARIES}
)

But yea, in essence you do have to tell both the sdk and your application.

[singku]

I assume you resolved this issue? @sunhyung

[sunhyung]

I assume you resolved this issue? @sunhyung

Yes, I closed the issue. For now, I'd rather choose editing platformDeps.cmake file. But if it would cause some problem later, I will consider what you guys explained to me.

Thank you for helping me a lot. :)