Open csi-amolpawar opened 1 week ago
Thanks for the detailed repro code and explanation. Looking into what might be causing this to fail.
Here is what I'm seeing in the logs
[ERROR] 2024-06-27 21:25:17.139 http-connection [140737202796288] static: Client connection failed with error 1029 (AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE).
[WARN] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failed to obtain new connection from http layer, error 1029(TLS (SSL) negotiation failed)
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failing excess connection acquisition with error code 1029
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] 2024-06-27 21:25:17.140 connection-manager [140737202796288] id=0x9dd080: Failed to complete connection acquisition with error_code 1029(TLS (SSL) negotiation failed)
[ERROR] 2024-06-27 21:25:17.140 S3Endpoint [140737202796288] id=0xb0a170: Could not acquire connection due to error code 1029 (TLS (SSL) negotiation failed)
It looks like config.caPath
might not be finding the correct file. For now you should be able to use config.caFile
to workaround this:
Aws::String ca_file = ca_path + "/cert.pem";
config.caFile = Aws::String(ca_file);
@jmklix It works fine when config.ca_file
is set. See
[root@bc281d03816c aws]# ./aws_s3_crt_w_ca_file
ca_file=/etc/pki/tls/cert.pem
object_content= 'This is a test file'
Previously, we were setting only ca_path
to default system path and was working as expected. See code snippet below
auto tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient();
tlsCtxOptions.SetVerifyPeer(!cfg->get<bool>(CTX_SKIP_VERIFY_SSL));
tlsCtxOptions.OverrideDefaultTrustStore(ca_path.c_str(), "");
...
auto tlsContext = Aws::Crt::Io::TlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
config.tlsConnectionOptions = Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>("libcsi-io-s3-crt", tlsContext.NewConnectionOptions());
and should be expected to work only setting directly in Aws::S3Crt::ClientConfiguration::ca_path
can you print out contents of your "/etc/pki/tls" folder?
For context, s3 crt uses s2n on linux for tls support (https://github.com/aws/s2n-tls) and s2n just uses libcrypto's X509_STORE_load_locations under the covers to load all the certs. So somehow libcrypto is not able to load certs from that path. Im not super familiar with where RHEL expects to store certs, but initial guess is that certs are in a slightly different folder or process somehow is not able to access them
Please see below
@DmitriyMusatkin
Describe the bug
Unable to do getobject when explicit 'Aws::S3Crt::ClientConfiguration::ca_path' is set to default path
Expected Behavior
Get Object should work as expected when ca_path is set explicitly.
When we don't set ca_path explicitly, it work fine.
Current Behavior
Receives the error message
GetObject error:TLS (SSL) negotiation failed (aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE)
Reproduction Steps
The issue is easily reproducible with below code snippet
Please note that
get_default_openssl_dir()
is evaluate to/etc/pki/tls
Possible Solution
NA
Additional Information/Context
Build Command:
g++ -std=c++20 -o <output> test_s3_crt_ca_path.cpp -I${AWS_INSTALL_PATH}/include -L${AWS_INSTALL_PATH}/lib64 -lcurl -lssl -lpthread -lcrypto -laws-cpp-sdk-s3-crt -laws-cpp-sdk-core
AWS CPP SDK version used
AWS SDK for C++ 1.11.351
Compiler and Version used
g++ (GCC) 13.2.0
Operating System and version
Red Hat Enterprise Linux 9.4 (Plow)