search

aws / aws-sdk-cpp

AWS SDK for C++
Apache License 2.0
1.97k stars 1.06k forks source link

Aws::Http::CurlHttpClient::MakeRequest triggers libcrypto crash #2027

Closed mcgrath-michael closed 3 months ago

mcgrath-michael commented 2 years ago

Describe the bug

Attempts to download from S3 cause a SIGPIPE exception despite httpOptions.installSigPipeHandler = true in Aws::SDKOptions passed to InitAPI. This is similar to issue https://github.com/aws/aws-sdk-cpp/issues/1534 , but we needed to set enableTcpKeepAlive = false in the S3 Client configuration to prevent the crash from occurring.

Expected Behavior

Expected httpOptions.installSigPipeHandler = true to be sufficient to prevent crash.

Current Behavior

The environment is aws-sdk-cpp 1.9.315 openssl 1.1.1n-0+deb11u3 libcurl4-openssl-dev:amd64 7.74.0-1.3+deb11u1 Debian 11 bullseye

The SIGPIPE exception occurs when an additional download is attempted after the system has been idle for a couple of minutes. The exception occurs during the CurlHttpClient::MakeRequest call to curl_easy_perform.

#0  __libc_write (nbytes=31, buf=0x7f9b14008f23, fd=97) at ../sysdeps/unix/sysv/linux/write.c:26
        resultvar = 18446744073709551584
        sc_cancel_oldtype = 0
        __arg3 = <optimized out>
        _a2 = <optimized out>
        sc_ret = <optimized out>
        __value = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        resultvar = <optimized out>
        resultvar = <optimized out>
        __arg2 = <optimized out>
        _a1 = <optimized out>
#1  __libc_write (fd=97, buf=0x7f9b14008f23, nbytes=31) at ../sysdeps/unix/sysv/linux/write.c:24
No locals.
#2  0x00007f9bf2cbc5c5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#3  0x00007f9bf2cb772a in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#4  0x00007f9bf2cb6784 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#5  0x00007f9bf2cb6c43 in BIO_write () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#6  0x00007f9bf2b8891a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#7  0x00007f9bf2b89809 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#8  0x00007f9bf2b931da in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#9  0x00007f9bf2b91425 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#10 0x00007f9beffda492 in ossl_close (connssl=<optimized out>, connssl=<optimized out>) at vtls/openssl.c:1365
        backend = 0x7f9b24003210
#11 0x00007f9beffda4f1 in Curl_ossl_close (conn=0x7f9b240093c0, sockindex=<optimized out>) at vtls/openssl.c:1382
No locals.
#12 0x00007f9beffe0b8b in Curl_ssl_close (conn=conn@entry=0x7f9b240093c0, sockindex=sockindex@entry=0) at vtls/vtls.c:632
No locals.
#13 0x00007f9beff96f4a in conn_shutdown (conn=0x7f9b240093c0) at url.c:720
No locals.
#14 Curl_disconnect (data=data@entry=0x7f9b9c008e50, conn=0x7f9b240093c0, dead_connection=dead_connection@entry=true) at url.c:838
No locals.
#15 0x00007f9beff99a8e in prune_dead_connections (data=0x7f9b9c008e50) at url.c:1040
        prune = {data = 0x7f9b9c008e50, extracted = 0x7f9b240093c0}
        now = {tv_sec = <optimized out>, tv_usec = <optimized out>}
        elapsed = <optimized out>
#16 create_conn (async=0x7f9bcb7f9717, in_connect=<synthetic pointer>, data=0x7f9bcb7f9690) at url.c:3725
        conn = <optimized out>
        conn_temp = 0x0
        reuse = <optimized out>
        connections_available = true
        force_reuse = false
        max_host_connections = <optimized out>
        result = <optimized out>
        waitpipe = false
        max_total_connections = <optimized out>
        result = <optimized out>
        conn = <optimized out>
        conn_temp = <optimized out>
        reuse = <optimized out>
        connections_available = <optimized out>
        force_reuse = <optimized out>
        waitpipe = <optimized out>
        max_host_connections = <optimized out>
        max_total_connections = <optimized out>
        out = <optimized out>
        done = <optimized out>
        bundlehost = <optimized out>
        bundle = <optimized out>
        conn_candidate = <optimized out>
        conn_candidate = <optimized out>
#17 Curl_connect (data=data@entry=0x7f9b9c008e50, asyncp=asyncp@entry=0x7f9bcb7f9787, protocol_done=protocol_done@entry=0x7f9bcb7f9788) at url.c:4003
        result = <optimized out>
        conn = <optimized out>
#18 0x00007f9beffb5c99 in multi_runsingle (multi=0x7f9b9c00a850, nowp=0x7f9bcb7f9800, data=0x7f9b9c008e50) at multi.c:1679
        stream_error = false
        msg = <optimized out>
        connected = false
        async = false
        protocol_connected = false
        dophase_done = false
        done = false
        rc = CURLM_OK
        result = CURLE_OK
        timeout_ms = <optimized out>
        recv_timeout_ms = <optimized out>
        send_timeout_ms = <optimized out>
        control = 32667
#19 0x00007f9beffb6c06 in curl_multi_perform (multi=multi@entry=0x7f9b9c00a850, running_handles=running_handles@entry=0x7f9bcb7f9908) at multi.c:2417
        result = <optimized out>
        pipe_st = {old_pipe_act = {__sigaction_handler = {sa_handler = 0x7f9bcb7f9860, sa_sigaction = 0x7f9bcb7f9860}, sa_mask = {__val = {8, 1, 140307741137729, 140307110795439, 140306313940560, 140306313947216, 140307722995779, 0, 511101108315, 237315, 468276273, 140306313940560, 468276, 140306313945576, 140307722881531, 237315}}, sa_flags = 1172939264, sa_restorer = 0x10}, no_signal = true}
        data = 0x7f9b9c008e50
        returncode = CURLM_OK
        t = 0x7f9bcb7f9860
        now = {tv_sec = 237315, tv_usec = 468302}
#20 0x00007f9beff8d943 in easy_transfer (multi=<optimized out>) at easy.c:593
        still_running = 0
        done = false
        mcode = <optimized out>
        result = CURLE_OK
        done = <optimized out>
        mcode = <optimized out>
        result = <optimized out>
        still_running = <optimized out>
        rc = <optimized out>
        msg = <optimized out>
#21 easy_perform (events=false, data=0x7f9b9c008e50) at easy.c:683
        multi = 0x7f9b9c00a850
        mcode = -880830200
        result = CURLE_OK
        pipe_st = {old_pipe_act = {__sigaction_handler = {sa_handler = 0x3000000018, sa_sigaction = 0x3000000018}, sa_mask = {__val = {140307110795760, 140307110795568, 15025353949859914240, 0, 0, 140307110795624, 140304032217600, 0, 140307110795696, 140307110795648, 140724835535128, 140307110795696, 0, 0, 140307741137729, 140307751789888}}, sa_flags = -880829984, sa_restorer = 0x7f9bf183bbd2 <Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const>}, no_signal = true}
        multi = <optimized out>
        mcode = <optimized out>
        result = <optimized out>
        pipe_st = {old_pipe_act = {__sigaction_handler = {sa_handler = <optimized out>, sa_sigaction = <optimized out>}, sa_mask = {__val = {<optimized out> <repeats 16 times>}}, sa_flags = <optimized out>, sa_restorer = <optimized out>}, no_signal = <optimized out>}
#22 curl_easy_perform (data=0x7f9b9c008e50) at easy.c:702
No locals.
#23 0x00007f9bf183ca0b in Aws::Http::CurlHttpClient::MakeRequest (this=0x22c4490, request=std::shared_ptr<Aws::Http::HttpRequest> (use count 2, weak count 0) = {...}, readLimiter=0x0, writeLimiter=0x0) at /aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHttpClient.cpp:731
        writeContext = {m_client = 0x22c4490, m_request = 0x7f9b14001150, m_response = 0x7f9b14003a00, m_rateLimiter = 0x0, m_numBytesResponseReceived = 0}
        readContext = {m_client = 0x22c4490, m_curlHandle = 0x7f9b9c008e50, m_rateLimiter = 0x0, m_request = 0x7f9b14001150, m_chunkEnd = false}
        startTransmissionTime = {m_time = {__d = {__r = 1660139662533519406}}, m_valid = true}
        curlResponseCode = 32667
        shouldContinueRequest = 20
        timep = 0
        ret = 32667
        curlGetInfoResult = 4052484852
        ip = 0x0
        uri = {m_scheme = Aws::Http::Scheme::HTTPS, m_authority = "s3.us-east-2.amazonaws.com", m_port = 443, m_pathSegments = std::vector of length 2, capacity 2 = {"forge.package.internal", "spark-task-bridge-plugin_0.9.2.fpk"}, m_pathHasTrailingSlash = false, m_queryString = ""}
        url = "https://s3.us-east-2.amazonaws.com/forge.package.internal/spark-task-bridge-plugin_0.9.2.fpk"
        response = std::shared_ptr<Aws::Http::HttpResponse> (use count 1, weak count 0) = {get() = 0x7f9b14003a00}
        headers = 0x7f9b140025c0
        headerStream = <incomplete type>
        requestHeaders = std::map with 9 elements = {["amz-sdk-invocation-id"] = "84DE2C92-1F9F-4C78-A8E5-29907DAC6B10", ["amz-sdk-request"] = "attempt=1", ["authorization"] = "AWS4-HMAC-SHA256 Credential=AKIAI3ZU2MK4YBOH6SXA/20220810/us-east-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-"..., ["content-type"] = "application/xml", ["host"] = "s3.us-east-2.amazonaws.com", ["user-agent"] = "aws-sdk-cpp/1.9.315 Linux/5.4.0-122-generic x86_64 GCC/10.2.1", ["x-amz-api-version"] = "2006-03-01", ["x-amz-content-sha256"] = "UNSIGNED-PAYLOAD", ["x-amz-date"] = "20220810T135422Z"}
        connectionHandle = 0x7f9b9c008e50
#24 0x00007f9bf189d0cb in Aws::Client::AWSClient::AttemptOneRequest (this=0x22c4910, httpRequest=std::shared_ptr<Aws::Http::HttpRequest> (use count 2, weak count 0) = {...}, request=..., signerName=0x7f9bcb7fbea0 "SignatureV4", signerRegionOverride=0x7f9bcb7fbe60 "us-east-2", signerServiceNameOverride=0x7f9bcb7fbe80 "s3") at /aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:506
        signer = 0x22c37d0
        httpResponse = std::shared_ptr<Aws::Http::HttpResponse> (use count 1, weak count 0) = {get() = 0x7f9b14003a00}
#25 0x00007f9bf189aa2b in Aws::Client::AWSClient::AttemptExhaustively (this=0x22c4910, uri=..., request=..., method=Aws::Http::HttpMethod::HTTP_HEAD, signerName=0x7f9bcb7fbea0 "SignatureV4", signerRegionOverride=0x7f9bcb7fbe60 "us-east-2", signerServiceNameOverride=0x7f9bcb7fbe80 "s3") at /aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:263
        serverTime = {m_time = {__d = {__r = 140307744171680}}, m_valid = false}
        retryWithCorrectRegion = false
        httpResponseCode = 0
        sleepMillis = 140307744171776
        newUri = {m_scheme = (unknown: 0x16), m_authority = <error reading variable: Cannot create a lazy string with address 0x0, and a non-zero length.>, m_port = 48408, m_pathSegments = std::vector of length 0, capacity 0, m_pathHasTrailingSlash = 96, m_queryString = "86_64 GCC/10.2.1us-east-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-type;host;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=04c5c474cdbc13bdb3ccf9964d"...}
        newEndpoint = <error: Cannot access memory at address 0x2d747361652d7375>
        clockSkew = {__r = 140307744171664}
        shouldSleep = false
        retries = 0
        httpRequest = std::shared_ptr<Aws::Http::HttpRequest> (use count 2, weak count 0) = {get() = 0x7f9b14001150}
        outcome = {result = std::shared_ptr<Aws::Http::HttpResponse> (empty) = {get() = 0x0}, error = {m_errorType = Aws::Client::CoreErrors::INCOMPLETE_SIGNATURE, m_exceptionName = "", m_message = "", m_remoteHostIpAddress = "", m_requestId = "", m_responseHeaders = std::map with 0 elements, m_responseCode = Aws::Http::HttpResponseCode::REQUEST_NOT_MADE, m_isRetryable = false, m_errorPayloadType = Aws::Client::ErrorPayloadType::NOT_SET, m_xmlPayload = {m_doc = 0x0}, m_jsonPayload = {m_value = 0x0, m_wasParseSuccessful = true, m_errorMessage = ""}}, success = false}
        lastError = {m_errorType = Aws::Client::CoreErrors::INCOMPLETE_SIGNATURE, m_exceptionName = "", m_message = "", m_remoteHostIpAddress = "", m_requestId = "", m_responseHeaders = std::map with 0 elements, m_responseCode = Aws::Http::HttpResponseCode::REQUEST_NOT_MADE, m_isRetryable = false, m_errorPayloadType = Aws::Client::ErrorPayloadType::NOT_SET, m_xmlPayload = {m_doc = 0x0}, m_jsonPayload = {m_value = 0x0, m_wasParseSuccessful = true, m_errorMessage = ""}}
        coreMetrics = {httpClientMetrics = std::map with 0 elements}
        contexts = std::vector of length 0, capacity 0
        signerRegion = 0x7f9bcb7fbe60 "us-east-2"
        regionFromResponse = ""
        invocationId = "84DE2C92-1F9F-4C78-A8E5-29907DAC6B10"
        requestInfo = {ttl = {m_time = {__d = {__r = 0}}, m_valid = true}, attempt = 1, maxAttempts = 0}
#26 0x00007f9bf18a22fb in Aws::Client::AWSXMLClient::MakeRequest (this=0x22c4910, uri=..., request=..., method=Aws::Http::HttpMethod::HTTP_HEAD, signerName=0x7f9bcb7fbea0 "SignatureV4", signerRegionOverride=0x7f9bcb7fbe60 "us-east-2", signerServiceNameOverride=0x7f9bcb7fbe80 "s3") at /aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:1227
        httpOutcome = {result = <error reading variable: Cannot access memory at address 0x7f9b00000020>, error = {m_errorType = 335548272, m_exceptionName = <error: Cannot access memory at address 0x22>, m_message = "", m_remoteHostIpAddress = "", m_requestId = "", m_responseHeaders = std::map with 140724835442207 elements<error reading variable: Cannot access memory at address 0x6c616e726584>, m_responseCode = -880814016, m_isRetryable = 155, m_errorPayloadType = (unknown: 0x802000), m_xmlPayload = {m_doc = 0x7f9bf4e106a0 <__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::deallocate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, unsigned long)+32>}, m_jsonPayload = {m_value = 0x7ffd0dd3de1f, m_wasParseSuccessful = true, m_errorMessage = ""}}, success = 144}
#27 0x00007f9bf20e6df3 in Aws::S3::S3Client::HeadObject (this=0x22c4910, request=...) at /aws-sdk-cpp/aws-cpp-sdk-s3/source/S3Client.cpp:2264
        computeEndpointOutcome = {result = {endpoint = "https://s3.us-east-2.amazonaws.com/forge.package.internal", signerRegion = "us-east-2", signerServiceName = "s3", signerName = "SignatureV4"}, error = {m_errorType = Aws::S3::S3Errors::INCOMPLETE_SIGNATURE, m_exceptionName = "", m_message = "", m_remoteHostIpAddress = "", m_requestId = "", m_responseHeaders = std::map with 0 elements, m_responseCode = Aws::Http::HttpResponseCode::REQUEST_NOT_MADE, m_isRetryable = false, m_errorPayloadType = Aws::Client::ErrorPayloadType::NOT_SET, m_xmlPayload = {m_doc = 0x0}, m_jsonPayload = {m_value = 0x0, m_wasParseSuccessful = true, m_errorMessage = ""}}, success = true}
        uri = {m_scheme = Aws::Http::Scheme::HTTPS, m_authority = "s3.us-east-2.amazonaws.com", m_port = 443, m_pathSegments = std::vector of length 2, capacity 2 = {"forge.package.internal", "spark-task-bridge-plugin_0.9.2.fpk"}, m_pathHasTrailingSlash = false, m_queryString = ""}
#28 0x00007f9bf2522865 in Aws::Transfer::TransferManager::InitializePartsForDownload (this=0x22c7de0, handle=std::shared_ptr<Aws::Transfer::TransferHandle> (use count 2, weak count 0) = {...}) at /aws-sdk-cpp/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp:743
        headObjectRequest = {<Aws::S3::S3Request> = {<Aws::AmazonSerializableWebServiceRequest> = {<Aws::AmazonWebServiceRequest> = {_vptr.AmazonWebServiceRequest = 0x7f9bf249d518 <vtable for Aws::S3::Model::HeadObjectRequest+16>, m_responseStreamFactory = {<std::_Maybe_unary_or_binary_function<std::basic_iostream<char, std::char_traits<char> >*>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f9bf18baf0c <Aws::Utils::Stream::DefaultResponseStreamFactoryMethod()>, _M_const_object = 0x7f9bf18baf0c <Aws::Utils::Stream::DefaultResponseStreamFactoryMethod()>, _M_function_pointer = 0x7f9bf18baf0c <Aws::Utils::Stream::DefaultResponseStreamFactoryMethod()>, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f9bf18baf0c <Aws::Utils::Stream::DefaultResponseStreamFactoryMethod()>}, _M_pod_data = "\f\257\213\361\233\177\000\000\000\000\000\000\000\000\000"}, _M_manager = 0x7f9bf18242be <std::_Function_handler<std::iostream* (), std::iostream* (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation)>}, _M_invoker = 0x7f9bf182429c <std::_Function_handler<std::iostream* (), std::iostream* (*)()>::_M_invoke(std::_Any_data const&)>}, m_onDataReceived = {<std::_Maybe_unary_or_binary_function<void, Aws::Http::HttpRequest const*, Aws::Http::HttpResponse*, long long>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}, m_onDataSent = {<std::_Maybe_unary_or_binary_function<void, Aws::Http::HttpRequest const*, long long>> = {<std::binary_function<Aws::Http::HttpRequest const*, long long, void>> = {<No data fields>}, <No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}, m_continueRequest = {<std::_Maybe_unary_or_binary_function<bool, Aws::Http::HttpRequest const*>> = {<std::unary_function<Aws::Http::HttpRequest const*, bool>> = {<No data fields>}, <No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}, m_onRequestSigned = {<std::_Maybe_unary_or_binary_function<void, Aws::Http::HttpRequest const&>> = {<std::unary_function<Aws::Http::HttpRequest const&, void>> = {<No data fields>}, <No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}, m_requestRetryHandler = {<std::_Maybe_unary_or_binary_function<void, Aws::AmazonWebServiceRequest const&>> = {<std::unary_function<Aws::AmazonWebServiceRequest const&, void>> = {<No data fields>}, <No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}}, <No data fields>}, <No data fields>}, m_bucket = "forge.package.internal", m_bucketHasBeenSet = true, m_ifMatch = "", m_ifMatchHasBeenSet = false, m_ifModifiedSince = {m_time = {__d = {__r = 0}}, m_valid = true}, m_ifModifiedSinceHasBeenSet = false, m_ifNoneMatch = "", m_ifNoneMatchHasBeenSet = false, m_ifUnmodifiedSince = {m_time = {__d = {__r = 0}}, m_valid = true}, m_ifUnmodifiedSinceHasBeenSet = false, m_key = "spark-task-bridge-plugin_0.9.2.fpk", m_keyHasBeenSet = true, m_range = "", m_rangeHasBeenSet = false, m_versionId = "", m_versionIdHasBeenSet = false, m_sSECustomerAlgorithm = "", m_sSECustomerAlgorithmHasBeenSet = false, m_sSECustomerKey = "", m_sSECustomerKeyHasBeenSet = false, m_sSECustomerKeyMD5 = "", m_sSECustomerKeyMD5HasBeenSet = false, m_requestPayer = Aws::S3::Model::RequestPayer::NOT_SET, m_requestPayerHasBeenSet = false, m_partNumber = 0, m_partNumberHasBeenSet = false, m_expectedBucketOwner = "", m_expectedBucketOwnerHasBeenSet = false, m_checksumMode = Aws::S3::Model::ChecksumMode::NOT_SET, m_checksumModeHasBeenSet = false, m_customizedAccessLogTag = std::map with 0 elements, m_customizedAccessLogTagHasBeenSet = true}
        headObjectOutcome = {result = {m_deleteMarker = false, m_acceptRanges = "", m_expiration = "", m_restore = "", m_archiveStatus = Aws::S3::Model::ArchiveStatus::NOT_SET, m_lastModified = {m_time = {__d = {__r = 0}}, m_valid = false}, m_contentLength = 0, m_checksumCRC32 = "", m_checksumCRC32C = "", m_checksumSHA1 = "", m_checksumSHA256 = "", m_eTag = "", m_missingMeta = 0, m_versionId = "", m_cacheControl = "", m_contentDisposition = "", m_contentEncoding = "", m_contentLanguage = "", m_contentType = "", m_expires = {m_time = {__d = {__r = 0}}, m_valid = false}, m_websiteRedirectLocation = "", m_serverSideEncryption = Aws::S3::Model::ServerSideEncryption::NOT_SET, m_metadata = std::map with 0 elements, m_sSECustomerAlgorithm = "", m_sSECustomerKeyMD5 = "", m_sSEKMSKeyId = "", m_bucketKeyEnabled = false, m_storageClass = Aws::S3::Model::StorageClass::NOT_SET, m_requestCharged = Aws::S3::Model::RequestCharged::NOT_SET, m_replicationStatus = Aws::S3::Model::ReplicationStatus::NOT_SET, m_partsCount = 0, m_objectLockMode = Aws::S3::Model::ObjectLockMode::NOT_SET, m_objectLockRetainUntilDate = {m_time = {__d = {__r = 0}}, m_valid = false}, m_objectLockLegalHoldStatus = Aws::S3::Model::ObjectLockLegalHoldStatus::NOT_SET}, error = {<Aws::Client::AWSError<Aws::S3::S3Errors>> = {m_errorType = Aws::S3::S3Errors::INCOMPLETE_SIGNATURE, m_exceptionName = "", m_message = "", m_remoteHostIpAddress = "", m_requestId = "", m_responseHeaders = std::map with 0 elements, m_responseCode = 0, m_isRetryable = false, m_errorPayloadType = Aws::Client::ErrorPayloadType::NOT_SET, m_xmlPayload = {m_doc = 0x0}, m_jsonPayload = {m_value = 0x0, m_wasParseSuccessful = false, m_errorMessage = ""}}, <No data fields>}, success = false}
        downloadSize = 0
        partCount = 0
        isRetry = false
        bufferSize = 5242880
#29 0x00007f9bf252322b in Aws::Transfer::TransferManager::DoDownload (this=0x22c7de0, handle=std::shared_ptr<Aws::Transfer::TransferHandle> (use count 2, weak count 0) = {...}) at /aws-sdk-cpp/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp:797
        isMultipart = false
        bufferSize = 140307110810784
        queuedParts = std::map with 0 elements
        queuedPartIter = non-dereferenceable iterator for associative container
#30 0x00007f9bf251c49e in operator() (__closure=0x7f9b9c0da1d0) at /aws-sdk-cpp/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp:107
        handle = std::shared_ptr<Aws::Transfer::TransferHandle> (use count 2, weak count 0) = {get() = 0x7f9b9c045bb0}
        self = warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<Aws::Transfer::TransferManager::Create(Aws::Transfer::TransferManagerConfiguration const&)::MakeSharedEnabler, std::allocator<Aws::Transfer::TransferManager::Create(Aws::Transfer::TransferManagerConfiguration const&)::MakeSharedEnabler>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<Aws::Transfer::TransferManager::Create(Aws::Transfer::TransferManagerConfiguration const&)::MakeSharedEnabler, std::allocator<Aws::Transfer::TransferManager::Create(Aws::Transfer::TransferManagerConfiguration const&)::MakeSharedEnabler>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr<Aws::Transfer::TransferManager> (use count 2, weak count 1) = {get() = 0x22c7de0}
#31 0x00007f9bf252fe50 in std::__invoke_impl<void, Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>&>(std::__invoke_other, struct {...} &) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
No locals.
#32 0x00007f9bf252fd38 in std::__invoke<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>&>(struct {...} &) (__fn=...) at /usr/include/c++/10/bits/invoke.h:95
No locals.
#33 0x00007f9bf252fb54 in std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()>::__call<void>(std::tuple<> &&, std::_Index_tuple<>) (this=0x7f9b9c0da1d0, __args=...) at /usr/include/c++/10/functional:416
No locals.
#34 0x00007f9bf252f850 in std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()>::operator()<>(void) (this=0x7f9b9c0da1d0) at /usr/include/c++/10/functional:499
No locals.
#35 0x00007f9bf252eb0c in std::__invoke_impl<void, std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()>&>(std::__invoke_other, std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()> &) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
No locals.
#36 0x00007f9bf252c666 in std::__invoke_r<void, std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()>&>(std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()> &) (__fn=...) at /usr/include/c++/10/bits/invoke.h:110
No locals.
#37 0x00007f9bf25297e1 in std::_Function_handler<void(), std::_Bind<Aws::Transfer::TransferManager::DownloadFile(const String&, const String&, Aws::Transfer::CreateDownloadStreamCallback, const Aws::Transfer::DownloadConfiguration&, const String&, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&)::<lambda()>()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
No locals.
#38 0x00007f9bf52f0f46 in std::function<void ()>::operator()() const () from /usr/local/lib/libzmqpp.so
No symbol table info available.
#39 0x00007f9bf18bc70d in Aws::Utils::Threading::ThreadTask::MainTaskRunner (this=0x226bd60) at /aws-sdk-cpp/aws-cpp-sdk-core/source/utils/threading/ThreadTask.cpp:31
        fn = 0x7f9b9c350980
#40 0x00007f9bf18f7b02 in std::__invoke_impl<void, void (Aws::Utils::Threading::ThreadTask::*&)(), Aws::Utils::Threading::ThreadTask*&> (__f=@0x22c7af8: (void (Aws::Utils::Threading::ThreadTask::*)(Aws::Utils::Threading::ThreadTask * const)) 0x7f9bf18bc692 <Aws::Utils::Threading::ThreadTask::MainTaskRunner()>, __t=@0x22c7b08: 0x226bd60) at /usr/include/c++/10/bits/invoke.h:73
No locals.
#41 0x00007f9bf18f79fd in std::__invoke<void (Aws::Utils::Threading::ThreadTask::*&)(), Aws::Utils::Threading::ThreadTask*&> (__fn=@0x22c7af8: (void (Aws::Utils::Threading::ThreadTask::*)(Aws::Utils::Threading::ThreadTask * const)) 0x7f9bf18bc692 <Aws::Utils::Threading::ThreadTask::MainTaskRunner()>) at /usr/include/c++/10/bits/invoke.h:95
No locals.
#42 0x00007f9bf18f786e in std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x22c7af8, __args=...) at /usr/include/c++/10/functional:416
No locals.
#43 0x00007f9bf18f76e4 in std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>::operator()<, void>() (this=0x22c7af8) at /usr/include/c++/10/functional:499
No locals.
#44 0x00007f9bf18f73c8 in std::__invoke_impl<void, std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>>(std::__invoke_other, std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>&&) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
No locals.
#45 0x00007f9bf18f70f3 in std::__invoke<std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>>(std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()>&&) (__fn=...) at /usr/include/c++/10/bits/invoke.h:95
No locals.
#46 0x00007f9bf18f6f1a in std::thread::_Invoker<std::tuple<std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x22c7af8) at /usr/include/c++/10/thread:264
No locals.
#47 0x00007f9bf18f6984 in std::thread::_Invoker<std::tuple<std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()> > >::operator()() (this=0x22c7af8) at /usr/include/c++/10/thread:271
No locals.
#48 0x00007f9bf18f6074 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (Aws::Utils::Threading::ThreadTask::*(Aws::Utils::Threading::ThreadTask*))()> > > >::_M_run() (this=0x22c7af0) at /usr/include/c++/10/thread:215
No locals.
#49 0x00007f9bf1305ed0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#50 0x00007f9bf26f1ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140307110815488, 3101955570438265997, 140724835442206, 140724835442207, 140307110811712, 8396800, -3153704086557926259, -3153723746508972915}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#51 0x00007f9bf1155def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Logs:

[2022-08-10 13:49:50.034][W][settings-service:1174541] [AWS:ClientConfiguration] Retry Strategy will use the default max attempts.
[2022-08-10 13:49:50.060][W][settings-service:1174541] [AWS:ClientConfiguration] Retry Strategy will use the default max attempts.
[2022-08-10 13:49:51.061][E][settings-service:1174541] [AWS:CurlHttpClient] Curl returned error code 28 - Timeout was reached
[2022-08-10 13:49:51.062][E][settings-service:1174541] [AWS:EC2MetadataClient] Http request to retrieve credentials failed
[2022-08-10 13:49:51.062][W][settings-service:1174541] [AWS:EC2MetadataClient] Request failed, now waiting 0 ms before attempting again.
[2022-08-10 13:49:52.063][E][settings-service:1174541] [AWS:CurlHttpClient] Curl returned error code 28 - Timeout was reached
[2022-08-10 13:49:52.063][E][settings-service:1174541] [AWS:EC2MetadataClient] Http request to retrieve credentials failed
[2022-08-10 13:49:52.063][E][settings-service:1174541] [AWS:EC2MetadataClient] Can not retrieve resource from http://169.254.169.254/latest/meta-data/placement/availability-zone

Reproduction Steps

See https://github.com/aws/aws-sdk-cpp/issues/1534 Download a file. Wait a minute. Download another file.

Possible Solution

Setting enableTcpKeepAlive = false in the S3 Client configuration prevented the crash from occurring.

Additional Information/Context

Build commands: mkdir sdk_build cd sdk_build cmake -S ../aws-sdk-cpp -B . -DCPP_STANDARD=17 -DENABLE_TESTING:BOOL=OFF -DCMAKE_BUILD_TYPE=Debug -DBUILD_ONLY="s3;transfer" cmake --build . sudo cmake --install .

AWS CPP SDK version used

1.9.315

Compiler and Version used

gcc (Debian 10.2.1-6) 10.2.1 20210110

Operating System and version

Debian 11 bullseye docker container running on Ubuntu 20.04

jmklix commented 4 months ago

Sorry for no reply on this initial issue. But we have made some changes to the CurlHttpClient since this issue was opened. I was unable to reproduce this issue with the current version of this sdk (1.11.367). Can you try updating to latest and see if you have any issues with this sdk?

github-actions[bot] commented 3 months ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.