search

aws / aws-iot-device-sdk-cpp-v2

Next generation AWS IoT Client SDK for C++ using the AWS Common Runtime
Apache License 2.0
185 stars 108 forks source link

Uses wrong libcrypto on wsl #756

Closed tech-meppem closed 4 weeks ago

tech-meppem commented 1 month ago

Describe the bug

I am building on wsl, using the -DUSE_OPENSSL=ON, as I am building for openwrt. We have a toolchain setup, and are trying to use the libs from the toolchain. It builds fine without -DUSE_OPENSSL, but then causes errors due to conflicts.

When running the command with -DUSE_OPENSSL, there are lots of errors with the cmake --build . --target install step.

I noticed that even though it prints out the correct paths in:

-- Found crypto: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.a  
-- LibCrypto Include Dir: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/include
-- LibCrypto Shared Lib:  /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.so
-- LibCrypto Static Lib:  /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.a

later it still prints out:

-- Using libcrypto from system: /mnt/c/msys64/mingw64/lib/libcrypto.dll.a

I have tried using -Dcrypto_INCLUDE_DIR & -Dcrypto_LIBRARY as well, they made no difference. And also CMAKE_IGNORE_PATH doesn't work.

I'm unsure what's likely to cause this disparity.

Also, I'm unsure if this is actually the thing causing all the errors or not, but it certainly is wrong.

Expected Behavior

It to use libcrypto from where it found it, not from the windows when using wsl.

Current Behavior

Prints

-- Using libcrypto from system: /mnt/c/msys64/mingw64/lib/libcrypto.dll.a

and many many errors in build output, such as:

/mnt/c/msys64/mingw64/include/stdlib.h:68:5: error: declaration for parameter ‘ldiv_t’ but no such parameter
   68 |   } ldiv_t;
      |     ^~~~~~
/mnt/c/msys64/mingw64/include/stdlib.h:63:5: error: declaration for parameter ‘div_t’ but no such parameter
   63 |   } div_t;
      |     ^~~~~
In file included from /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-common/include/aws/common/stdint.h:8,
                 from /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-common/include/aws/common/allocator.h:10,
                 from /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-common/include/aws/common/common.h:12,
                 from /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/include/aws/cal/cal.h:8,
                 from /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/cal.c:5:
/home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-common/include/aws/common/assert.h:18:6: error: declaration for parameter ‘aws_fatal_assert’ but no such parameter
   18 | void aws_fatal_assert(const char *cond_str, const char *file, int line) AWS_ATTRIBUTE_NORETURN;
      |      ^~~~~~~~~~~~~~~~
/home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/cal.c:112: error: expected ‘{’ at end of input
  112 | }
      | 
/home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/cal.c:112:1: warning: control reaches end of non-void function [-Wreturn-type]
  112 | }
      | ^
make[2]: *** [crt/aws-crt-cpp/crt/aws-c-cal/CMakeFiles/aws-c-cal.dir/build.make:63: crt/aws-crt-cpp/crt/aws-c-cal/CMakeFiles/aws-c-cal.dir/source/cal.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1527: crt/aws-crt-cpp/crt/aws-c-cal/CMakeFiles/aws-c-cal.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

Reproduction Steps

Command I am using:

cmake.cross file:

set(CMAKE_IGNORE_PATH "/mnt/c/msys64/**")
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER "/home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-gcc")
set(CMAKE_CXX_COMPILER "/home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-g++")
set(CMAKE_FIND_ROOT_PATH "/home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_SYSTEM_PROCESSOR "arm")

command:

    cmake -DCMAKE_INSTALL_PREFIX="$TOP_DIR/$SDK_DIR" \
        -DCMAKE_TOOLCHAIN_FILE="$TOP_DIR/cmake.cross" \
        -DUSE_OPENSSL=ON \
        -DCMAKE_BUILD_TYPE="Debug" \
        ../aws-iot-device-sdk-cpp-v2

output:

-- AWS SDK C++ 1.33.0-dev+dc12eb50
-- The C compiler identification is GNU 9.2.0
-- The CXX compiler identification is GNU 9.2.0
-- Check for working C compiler: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-gcc
-- Check for working C compiler: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-g++
-- Check for working CXX compiler: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- AWS CRT C++ 0.27.3
-- CMake 3.16.3
-- Performing Test AWS_HAVE_WINAPI_DESKTOP
-- Performing Test AWS_HAVE_WINAPI_DESKTOP - Failed
-- Performing Test AWS_ARCH_INTEL
-- Performing Test AWS_ARCH_INTEL - Failed
-- Performing Test AWS_ARCH_INTEL_X64
-- Performing Test AWS_ARCH_INTEL_X64 - Failed
-- Performing Test AWS_ARCH_ARM64
-- Performing Test AWS_ARCH_ARM64 - Failed
-- Performing Test AWS_ARCH_ARM32
-- Performing Test AWS_ARCH_ARM32 - Success
-- Performing Test AWS_HAVE_GCC_INLINE_ASM
-- Performing Test AWS_HAVE_GCC_INLINE_ASM - Success
-- Performing Test AWS_HAVE_AUXV
-- Performing Test AWS_HAVE_AUXV - Success
-- Performing Test AWS_HAVE_EXECINFO
-- Performing Test AWS_HAVE_EXECINFO - Success
-- Performing Test AWS_HAVE_LINUX_IF_LINK_H
-- Performing Test AWS_HAVE_LINUX_IF_LINK_H - Success
-- Performing Test HAVE_M_AVX2_FLAG
-- Performing Test HAVE_M_AVX2_FLAG - Failed
-- Performing Test HAVE_M_AVX512_FLAG
-- Performing Test HAVE_M_AVX512_FLAG - Failed
-- Performing Test AWS_HAVE_AVX2_INTRINSICS
-- Performing Test AWS_HAVE_AVX2_INTRINSICS - Failed
-- Performing Test AWS_HAVE_AVX512_INTRINSICS
-- Performing Test AWS_HAVE_AVX512_INTRINSICS - Failed
-- Performing Test AWS_HAVE_MM256_EXTRACT_EPI64
-- Performing Test AWS_HAVE_MM256_EXTRACT_EPI64 - Failed
-- Performing Test AWS_HAVE_CLMUL
-- Performing Test AWS_HAVE_CLMUL - Failed
-- Performing Test AWS_HAVE_ARM32_CRC
-- Performing Test AWS_HAVE_ARM32_CRC - Failed
-- Performing Test AWS_HAVE_ARMv8_1
-- Performing Test AWS_HAVE_ARMv8_1 - Failed
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Performing Test HAS_FPIC_FLAG
-- Performing Test HAS_FPIC_FLAG - Success
-- Performing Test HAS_64BIT_FILE_OFFSET_BY_DEFAULT
-- Performing Test HAS_64BIT_FILE_OFFSET_BY_DEFAULT - Failed
-- Performing Test HAS_64BIT_FILE_OFFSET_VIA_DEFINES
-- Performing Test HAS_64BIT_FILE_OFFSET_VIA_DEFINES - Success
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdbool.h
-- Looking for stdbool.h - found
-- Performing Test HAS_WGNU
-- Performing Test HAS_WGNU - Failed
-- Performing Test HAVE_SYSCONF
-- Performing Test HAVE_SYSCONF - Success
-- Looking for pthread_attr_setaffinity_np
-- Looking for pthread_attr_setaffinity_np - found
-- Performing Test PTHREAD_SETNAME_TAKES_2ARGS
-- Performing Test PTHREAD_SETNAME_TAKES_2ARGS - Success
-- Performing Test PTHREAD_GETNAME_TAKES_2ARGS
-- Performing Test PTHREAD_GETNAME_TAKES_2ARGS - Failed
-- Performing Test PTHREAD_GET_NAME_TAKES_2ARGS
-- Performing Test PTHREAD_GET_NAME_TAKES_2ARGS - Failed
-- Performing Test PTHREAD_GETNAME_TAKES_3ARGS
-- Performing Test PTHREAD_GETNAME_TAKES_3ARGS - Success
-- Packaging currently only supported on Fedora.
-- Detected CMAKE_SYSTEM_PROCESSOR as arm
-- Detected 32-Bit system
-- Found crypto: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.a  
-- LibCrypto Include Dir: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/include
-- LibCrypto Shared Lib:  /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.so
-- LibCrypto Static Lib:  /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/target-arm-buildroot-linux-gnueabi_glibc/usr/lib/libcrypto.a
-- Using libcrypto from the cmake path
-- CMAKE_AR found: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-ar
-- CMAKE_RANLIB found: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-ranlib
-- CMAKE_OBJCOPY found: /home/path/to/unit/unit-sdk/BRCM_SDK_5.04L02p1/altsdk/openwrt/v19.07.master/staging_dir/toolchain-arm-buildroot-linux-gnueabi/bin/arm-buildroot-linux-gnueabi-objcopy
-- feature S2N_ATOMIC_SUPPORTED: TRUE
-- feature S2N_CLOEXEC_SUPPORTED: TRUE
-- feature S2N_CLOEXEC_XOPEN_SUPPORTED: TRUE
-- feature S2N_CLONE_SUPPORTED: TRUE
-- feature S2N_CPUID_AVAILABLE: FALSE
-- feature S2N_DIAGNOSTICS_POP_SUPPORTED: TRUE
-- feature S2N_DIAGNOSTICS_PUSH_SUPPORTED: TRUE
-- feature S2N_EXECINFO_AVAILABLE: TRUE
-- feature S2N_FALL_THROUGH_SUPPORTED: TRUE
-- feature S2N_FEATURES_AVAILABLE: TRUE
-- feature S2N_KTLS_SUPPORTED: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_EC_KEY_CHECK_FIPS: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_AEAD_TLS: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_MD_CTX_SET_PKEY_CTX: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_RC4: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_FLAG_NO_CHECK_TIME: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_HKDF: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_KYBER: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_RSA_PSS_SIGNING: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_X509_STORE_LIST: TRUE
-- feature S2N_LINUX_SENDFILE: TRUE
-- feature S2N_MADVISE_SUPPORTED: TRUE
-- feature S2N_MINHERIT_SUPPORTED: FALSE
-- feature S2N_STACKTRACE: TRUE
-- Using libcrypto from system: /mnt/c/msys64/mingw64/lib/libcrypto.dll.a
-- Performing Test AWS_ARM32_CRC
-- Performing Test AWS_ARM32_CRC - Success
IotJobs version is 1.33.0
IotShadow version is 1.33.0
Discovery version is 1.33.0
IotIdentity version is 1.33.0
EventStreamRPC version is 1.33.0
GreengrassIpc version is 1.33.0
IoT Device Common version is 1.33.0
Device Defender version is 1.33.0
Secure Tunneling version is 1.33.0
-- Configuring done
-- Generating done
-- Build files have been written to: /home/path/to/unit-claim-test/sdk-workspace/aws-iot-device-sdk-cpp-v2-build

Possible Solution

Edit: /mnt/c/msys64/mingw64/bin is in $PATH. I will try removing and testing if that works.

Edit 2: That has worked! I added this to my .bashrc file:

# remove `/mnt/c/msys64/mingw64/bin` from PATH
export PATH=$(echo $PATH | sed -e 's/:\/mnt\/c\/msys64\/mingw64\/bin//')

So I guess there's an issue where it can prioritise the $PATH over what was previously printed out, and specified using the arguments?

Additional Information/Context

No response

SDK version used

1.33.0

Environment details (OS name and version, etc.)

WSL version: 2.3.24.0 Kernel version: 5.15.153.1-2 WSLg version: 1.0.65 MSRDC version: 1.2.5620 Direct3D version: 1.611.1-81528511 DXCore version: 10.0.26100.1-240331-1435.ge-release Windows version: 10.0.19045.5011

tech-meppem commented 1 month ago

I found a solution: /mnt/c/msys64/mingw64/bin is in $PATH. Removing that has worked! I added this to my .bashrc file:

# remove `/mnt/c/msys64/mingw64/bin` from PATH
export PATH=$(echo $PATH | sed -e 's/:\/mnt\/c\/msys64\/mingw64\/bin//')

So I guess the issue is that it can prioritise the $PATH in different places, and doesn't prioritise it in others?

bretambrose commented 1 month ago

This appears to be more an issue of how Windows and WSL interact than an issue in the SDK.

tech-meppem commented 3 weeks ago

This appears to be more an issue of how Windows and WSL interact than an issue in the SDK.

For that specific case, windows+wsl explains it, sort of.

However, I think the directories in the PATH env shouldn't be prioritised over the individual directories passed in via args. We're now having issues with our build toolchain (not on WSL) due to this same problem, where it pulls the crypto lib from the PATH over pulling from any arguments passed in.

After all, if specifying -Dcrypto_LIBRARY= & -Dcrypto_INCLUDE_DIR=, it should probably always use those, not any directories in PATH.

Especially as it's not actually using the direct path in lib, but rather, an adjacent path. I had /mnt/c/msys64/mingw64/bin in my PATH env, but it was causing libcrypto to be linked to /mnt/c/msys64/mingw64/lib, which is a different path.