Error in installation: SAM build fails for Python39

mujtaba1747 commented 6 months ago

The documentation says that running sam build and sam deploy --guided will deploy the layer. But, I get an error when I followed this.

Steps to reproduce the issue:

Fork the fmi repository
Setup AWS credentials (access key and id)
cd python/aws/python39
Run sam build

The error message I got was:

Build Failed
Error: CustomMakeBuilder:MakeBuild - Make Failed: Unable to find image 'fmi-build-python39:latest' locally
docker: Error response from daemon: pull access denied for fmi-build-python39, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
make: *** [build-python39] Error 125

Possible way to fix this:

Update documentation to include step to build the docker image from this repo: https://github.com/OpenCoreCH/FMI-build-docker

Update on this: I fixed this by building the docker image from the repo

mujtaba1747 commented 6 months ago

Apart from this, I've also faced a few more errors and hiccups trying to build the lambda layer.

Once I figure it all out, I can contribute some documentation on how to install it step by step

mujtaba1747 commented 6 months ago

This is the latest error

Steps to reproduce:

Clone FMI repo
Clone docker build repo: https://github.com/OpenCoreCH/FMI-build-docker. Build the docker image for python 3.9
Clone TCPunch into extern/TCPunch (It is a git submodule)
Add this line to CMakelists.txt in the project root folder:
```
find_package(ZLIB REQUIRED)
```
Why did I add this line? --> Because, before adding this line, I kept getting this error in sam build

  Target "fmi" links to target "ZLIB::ZLIB" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?

cd python/aws/python39

On running sam build, I get this error. Probably related to some AWS S3 client library:


Building layer 'python39'                                                                    
python39: Running CustomMakeBuilder:CopySource                                               
python39: Running CustomMakeBuilder:MakeBuild                                                
python39: Current Artifacts Directory :                                                      
/workspace/fmi/python/aws/python39/.aws-sam/build/python39                                   
docker run -it --mount type=bind,source=/workspace/fmi/python/aws/python39//../../../,target=/opt/fmi/ --rm fmi-build-python39:latest bash -c "rm -rf /opt/fmi/python/build/* && mkdir -p /opt/fmi/python/build/ && cd /opt/fmi/python/build/ && cmake .. && make"
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /usr/local/lib/cmake/Boost-1.77.0/BoostConfig.cmake (found version "1.77.0") found components: log 
-- Found AWS SDK for C++, Version: 1.11.274, Install Root:/usr/local, Platform Prefix:, Platform Dependent Libraries: pthread;crypto;ssl;z;curl
-- Components specified for AWSSDK: s3, application will be depending on libs: aws-cpp-sdk-s3;aws-cpp-sdk-core;aws-crt-cpp;aws-c-auth;aws-c-cal;aws-c-common;aws-c-compression;aws-c-event-stream;aws-c-http;aws-c-io;aws-c-mqtt;aws-c-s3;aws-checksums;aws-c-sdkutils;pthread;crypto;ssl;z;curl
-- Try finding aws-cpp-sdk-core
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Found crypto: /usr/lib64/libcrypto.so  
-- LibCrypto Include Dir: /usr/include
-- LibCrypto Shared Lib:  /usr/lib64/libcrypto.so
-- LibCrypto Static Lib:  crypto_STATIC_LIBRARY-NOTFOUND
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.0.2k")  
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- S2N found target: AWS::crypto
-- crypto Include Dir: /usr/include
-- Found aws-cpp-sdk-core
-- Try finding aws-cpp-sdk-s3
-- Found aws-cpp-sdk-s3
-- Found hiredis: /usr/local/lib/libhiredis.a  
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") 
-- Found Boost: /usr/local/lib/cmake/Boost-1.77.0/BoostConfig.cmake (found version "1.77.0") found components: system python3 
-- Found Python3: /var/lang/bin/python3.9 (found version "3.9.6") found components: Interpreter Development Development.Module Development.Embed 
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/fmi/python/build
Scanning dependencies of target tcpunch
[  6%] Building CXX object libfmi/extern/TCPunch/client/CMakeFiles/tcpunch.dir/tcpunch.cpp.o
[ 13%] Linking CXX static library libtcpunch.a
[ 13%] Built target tcpunch
Scanning dependencies of target FMI
[ 20%] Building CXX object libfmi/CMakeFiles/FMI.dir/src/Communicator.cpp.o
[ 26%] Building CXX object libfmi/CMakeFiles/FMI.dir/src/utils/Configuration.cpp.o
[ 33%] Building CXX object libfmi/CMakeFiles/FMI.dir/src/comm/Channel.cpp.o
[ 40%] Building CXX object libfmi/CMakeFiles/FMI.dir/src/comm/ClientServer.cpp.o
[ 46%] Building CXX object libfmi/CMakeFiles/FMI.dir/src/comm/S3.cpp.o
In file included from /usr/local/include/aws/core/utils/memory/stl/AWSAllocator.h:11:0,
             from /usr/local/include/aws/core/utils/memory/stl/AWSString.h:10,
             from /usr/local/include/aws/core/utils/DateTime.h:9,
             from /usr/local/include/aws/core/auth/AWSCredentialsProvider.h:13,
             from /opt/fmi/src/comm/S3.cpp:1:
/usr/local/include/aws/core/utils/memory/AWSMemory.h: In instantiation of ‘T* Aws::New(const char*, ArgTypes&& ...) [with T = Aws::S3::S3Client; ArgTypes = {std::shared_ptr<Aws::Auth::EnvironmentAWSCredentialsProvider>&, Aws::Client::ClientConfiguration&}]’:
/usr/local/include/aws/core/utils/memory/AWSMemory.h:304:40:   required from ‘Aws::UniquePtr<T> Aws::MakeUnique(const char*, ArgTypes&& ...) [with T = Aws::S3::S3Client; ArgTypes = {std::shared_ptr<Aws::Auth::EnvironmentAWSCredentialsProvider>&, Aws::Client::ClientConfiguration&}; Aws::UniquePtr<T> = std::unique_ptr<Aws::S3::S3Client, Aws::Deleter<Aws::S3::S3Client> >]’
/opt/fmi/src/comm/S3.cpp:28:81:   required from here
/usr/local/include/aws/core/utils/memory/AWSMemory.h:67:32: error: no matching function for call to ‘Aws::S3::S3Client::S3Client(std::shared_ptr<Aws::Auth::EnvironmentAWSCredentialsProvider>&, Aws::Client::ClientConfiguration&)’
     T *constructedMemory = new (rawMemory) T(std::forward<ArgTypes>(args)...);
                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/fmi/src/comm/../../include/comm/S3.h:7:0,
             from /opt/fmi/src/comm/S3.cpp:2:
/usr/local/include/aws/s3/S3Client.h:129:9: note: candidate: Aws::S3::S3Client::S3Client(const std::shared_ptr<Aws::Auth::AWSCredentialsProvider>&, const Aws::Client::ClientConfiguration&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy, bool, Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION)
     S3Client(const std::shared_ptr<Aws::Auth::AWSCredentialsProvider>& credentialsProvider,
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:129:9: note:   candidate expects 5 arguments, 2 provided
/usr/local/include/aws/s3/S3Client.h:119:9: note: candidate: Aws::S3::S3Client::S3Client(const Aws::Auth::AWSCredentials&, const Aws::Client::ClientConfiguration&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy, bool, Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION)
     S3Client(const Aws::Auth::AWSCredentials& credentials,
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:119:9: note:   candidate expects 5 arguments, 2 provided
/usr/local/include/aws/s3/S3Client.h:110:9: note: candidate: Aws::S3::S3Client::S3Client(const Aws::Client::ClientConfiguration&, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy, bool, Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION)
     S3Client(const Aws::Client::ClientConfiguration& clientConfiguration,
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:110:9: note:   candidate expects 4 arguments, 2 provided
/usr/local/include/aws/s3/S3Client.h:100:9: note: candidate: Aws::S3::S3Client::S3Client(const std::shared_ptr<Aws::Auth::AWSCredentialsProvider>&, std::shared_ptr<Aws::Endpoint::EndpointProviderBase<Aws::S3::S3ClientConfiguration, Aws::S3::Endpoint::S3BuiltInParameters, Aws::S3::Endpoint::S3ClientContextParameters> >, const Aws::S3::S3ClientConfiguration&)
     S3Client(const std::shared_ptr<Aws::Auth::AWSCredentialsProvider>& credentialsProvider,
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:100:9: note:   no known conversion for argument 2 from ‘Aws::Client::ClientConfiguration’ to ‘std::shared_ptr<Aws::Endpoint::EndpointProviderBase<Aws::S3::S3ClientConfiguration, Aws::S3::Endpoint::S3BuiltInParameters, Aws::S3::Endpoint::S3ClientContextParameters> >’
/usr/local/include/aws/s3/S3Client.h:92:9: note: candidate: Aws::S3::S3Client::S3Client(const Aws::Auth::AWSCredentials&, std::shared_ptr<Aws::Endpoint::EndpointProviderBase<Aws::S3::S3ClientConfiguration, Aws::S3::Endpoint::S3BuiltInParameters, Aws::S3::Endpoint::S3ClientContextParameters> >, const Aws::S3::S3ClientConfiguration&)
     S3Client(const Aws::Auth::AWSCredentials& credentials,
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:92:9: note:   no known conversion for argument 1 from ‘std::shared_ptr<Aws::Auth::EnvironmentAWSCredentialsProvider>’ to ‘const Aws::Auth::AWSCredentials&’
/usr/local/include/aws/s3/S3Client.h:85:9: note: candidate: Aws::S3::S3Client::S3Client(const Aws::S3::S3ClientConfiguration&, std::shared_ptr<Aws::Endpoint::EndpointProviderBase<Aws::S3::S3ClientConfiguration, Aws::S3::Endpoint::S3BuiltInParameters, Aws::S3::Endpoint::S3ClientContextParameters> >)
     S3Client(const Aws::S3::S3ClientConfiguration& clientConfiguration = Aws::S3::S3ClientConfiguration(),
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:85:9: note:   no known conversion for argument 1 from ‘std::shared_ptr<Aws::Auth::EnvironmentAWSCredentialsProvider>’ to ‘const Aws::S3::S3ClientConfiguration&’
In file included from /opt/fmi/src/comm/../../include/comm/S3.h:7:0,
             from /opt/fmi/src/comm/S3.cpp:2:
/usr/local/include/aws/s3/S3Client.h:72:9: note: candidate: Aws::S3::S3Client::S3Client(Aws::S3::S3Client&&)
     S3Client(S3Client &&rhs);
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:72:9: note:   candidate expects 1 argument, 2 provided
/usr/local/include/aws/s3/S3Client.h:57:9: note: candidate: Aws::S3::S3Client::S3Client(const Aws::S3::S3Client&)
     S3Client(const S3Client &rhs);
     ^~~~~~~~
/usr/local/include/aws/s3/S3Client.h:57:9: note:   candidate expects 1 argument, 2 provided
make[2]: *** [libfmi/CMakeFiles/FMI.dir/src/comm/S3.cpp.o] Error 1
make[1]: *** [libfmi/CMakeFiles/FMI.dir/all] Error 2
make: *** [all] Error 2

Build Failed Error: CustomMakeBuilder:MakeBuild - Make Failed: make: *** [/workspace/fmi/python/aws/python39/Makefile:5: build-python39] Error 2

mujtaba1747 commented 6 months ago

It seems that the aws c++ sdk might have changed over the years. And now it doesn't support the constructor in line 28 in src/comm/S3.cpp

mujtaba1747 commented 6 months ago

I found a temporary fix to the error: Replace line 28 in src/comm/S3.cpp with client = Aws::MakeUnique<Aws::S3::S3Client>("S3Client", config) instead of the previous client = Aws::MakeUnique<Aws::S3::S3Client>(TAG, credentialsProvider, config);

The latest aws sdk includes a default credentialsProvider in the ClientConfiguration object. And passing a credentialsProvider as a function parameter resulted in the build error.

What is a permanent fix to this error:

Attach the credentialProvider in S3.cpp as a custom crentialProvider to ClientConfiguration config
Run tests to validate the fix

There is a small chance I might be wrong in my approach to the permanent fix because I haven't looked into how FMI accesses AWS credentials yet.

mujtaba1747 commented 6 months ago

The root cause of this error:

aws sdk for c++ had new updates over the years. And eventually, the newer version had a constructor for the S3 Client that was not backward compatible
The build scripts (both docker and local) cloned the latest version of aws sdk for c++ instead of a fixed version. But, I understand this decision because sometimes older libraries end up having security flaws.

mcopik commented 6 months ago

@mujtaba1747 Thank you for your debugging effort, this is very helpful!

Regarding ZLIB, this seems to be an easy error to fix - please feel free to open a PR with just this change.
Regarding the AWS credentials issues, I think you are right. FMI does not need the credentials to run; it only needs the AWS S3 and Dynamo clients to be properly configured. Again, please feel free to open a PR :)

mujtaba1747 commented 6 months ago

@mcopik Raised a PR for the ZLIB fix: https://github.com/spcl/fmi/pull/18

spcl / fmi

Error in installation: SAM build fails for Python39 #17