Closed lordgamez closed 6 days ago
yasminetalby
commented
1 year ago Hello @lordgamez ,
Thank you very much for your submission. Could you please provide the trace log associated with the behavior All you need to do is: options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace; (See for documentation on logging)?
It seems that the request signature issue you are encountering might be caused by a time skew.
Best regards,
Yasmine
lordgamez
commented
1 year ago Hi @yasminetalby,
Thank you for the quick response! I collected trace logs from an 8MB and a 12MB file upload: aws_failure_8MB.log aws_failure_12MB.log
In the meantime I tried building the library with the USE_CRT_HTTP_CLIENT=ON flag on version 1.11.118 to use the CRT HTTP client instead of the winhttp calls. I ran into this issue as I also set the server side encryption every time with the default value of NOT_SET and due to this the upload failed. After fixing the issue with the proposed workaround the upload succeeded on Windows with the CRT HTTP client. Do you think this could be a good temporary workaround until this issue is figured out? Which one is the preferred, more future proof approach, using the CRT HTTP client on all platforms or the platform specific WinHTTP on Windows and Curl on Linux?
BR, Gabor
lordgamez
commented
1 year ago Some additional info regarding the CRT HTTP client:
I tested it on Ubuntu 22.04 as well, and while the Curl backend constantly succeeds there seems to be random failures with the CRT client with the error Request Timeout Has Expired, see the attached trace logs for more information. Could this be due to some configuration issue?
Edit: The same issue appears randomly on Windows 10, additionally when using CRT HTTP lib on Win10 even when the main thread exits, the process keeps hanging and does not exist for some reason.
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint str eval parameter: Region = eu-west-2
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint bool eval parameter: UseFIPS = 0
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint bool eval parameter: UseDualStack = 0
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint bool eval parameter: UseArnRegion = 0
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint str eval parameter: Bucket = testbucket
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [Aws::Endpoint::DefaultEndpointProvider] Endpoint rules engine evaluated the endpoint: https://testbucket.s3.eu-west-2.amazonaws.com
[2023-07-18 14:21:23.413] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [Aws::Endpoint::DefaultEndpointProvider] Endpoint rules evaluated props: {"authSchemes":[{"disableDoubleEncoding":true,"name":"sigv4","signingName":"s3","signingRegion":"eu-west-2"}]}
[2023-07-18 14:21:23.414] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [AWSClient] Found body, but content-length has not been set, attempting to compute content-length
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Canonical Header String: amz-sdk-invocation-id:C5794CA0-4ED1-4CB3-BE2B-513134FCE44A
amz-sdk-request:attempt=1
content-length:15728640
content-md5:Tlv3ZX2UNZUYIlvmfZhw2Q==
content-type:application/octet-stream
host:testbucket.s3.eu-west-2.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230718T122123Z
x-amz-grant-full-control:
x-amz-grant-read:
x-amz-grant-read-acp:
x-amz-grant-write-acp:
x-amz-storage-class:STANDARD
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-storage-class
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Canonical Request String: PUT
/1689682882982900132
amz-sdk-invocation-id:C5794CA0-4ED1-4CB3-BE2B-513134FCE44A
amz-sdk-request:attempt=1
content-length:15728640
content-md5:Tlv3ZX2UNZUYIlvmfZhw2Q==
content-type:application/octet-stream
host:testbucket.s3.eu-west-2.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230718T122123Z
x-amz-grant-full-control:
x-amz-grant-read:
x-amz-grant-read-acp:
x-amz-grant-write-acp:
x-amz-storage-class:STANDARD
amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-storage-class
UNSIGNED-PAYLOAD
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Final String to sign: AWS4-HMAC-SHA256
20230718T122123Z
20230718/eu-west-2/s3/aws4_request
fa28ce5c5a03255d0b3387e632af022703b6e985be52b7c1511b3347edd0c500
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Final computed signing hash: 871e0fbfcada0cbef9f5deffc4ac7708247aca6caccdf98b2cf45a0a3f1472fc
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSAuthV4Signer] Signing request with: AWS4-HMAC-SHA256 Credential=AKIAUYLDB5SQVHSVDTJS/20230718/eu-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-storage-class, Signature=871e0fbfcada0cbef9f5deffc4ac7708247aca6caccdf98b2cf45a0a3f1472fc
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSClient] Request Successfully signed
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] Making PUT request to https://testbucket.s3.eu-west-2.amazonaws.com/1689682882982900132
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] Including headers:
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] amz-sdk-invocation-id: C5794CA0-4ED1-4CB3-BE2B-513134FCE44A
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] amz-sdk-request: attempt=1
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] authorization: AWS4-HMAC-SHA256 Credential=AKIAUYLDB5SQVHSVDTJS/20230718/eu-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-grant-read;x-amz-grant-read-acp;x-amz-grant-write-acp;x-amz-storage-class, Signature=871e0fbfcada0cbef9f5deffc4ac7708247aca6caccdf98b2cf45a0a3f1472fc
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] content-length: 15728640
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] content-md5: Tlv3ZX2UNZUYIlvmfZhw2Q==
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] content-type: application/octet-stream
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] host: testbucket.s3.eu-west-2.amazonaws.com
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] user-agent: aws-sdk-cpp/1.11.118 Linux/5.19.0-46-generic x86_64 GCC/11.3.0
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-content-sha256: UNSIGNED-PAYLOAD
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-date: 20230718T122123Z
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-grant-full-control:
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-grant-read:
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-grant-read-acp:
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-grant-write-acp:
[2023-07-18 14:21:23.431] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [trace] [CRTHttpClient] x-amz-storage-class: STANDARD
[2023-07-18 14:21:23.650] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [CRTHttpClient] Obtained connection handle 0x7f982c02a2e0
[2023-07-18 14:21:26.432] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSClient] Request returned error. Attempting to generate appropriate error codes from response
[2023-07-18 14:21:26.432] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [error] [AWSXmlClient] HTTP response code: -1
Resolved remote host IP address:
Request ID:
Exception name:
Error message: Request Timeout Has Expired
0 response headers:
[2023-07-18 14:21:26.432] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2023-07-18 14:21:26.432] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSClient] Date header was not found in the response, can't attempt to detect clock skew
[2023-07-18 14:21:26.433] [org::apache::nifi::minifi::aws::s3::S3RequestSender] [error] PutS3Object failed with the following: 'Request Timeout Has Expired'Hello @lordgamez ,
My apologies for the delay of answer. Thank you very much for providing the information requested. First of all let’s go over your first comment. I have gone over the log and we can see two different behavior here:
For the attempt to upload a 8MB file, the issue is caused by a difference between the server time and the client time :
[2023-07-18 11:12:11.823] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2023-07-18 11:12:11.823] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSClient] Server time is Tue, 18 Jul 2023 09:12:10 GMT, while client time is Tue, 18 Jul 2023 09:12:11 GMT
[2023-07-18 11:12:11.823] [org::apache::nifi::minifi::aws::s3::S3RequestSender] [error] PutS3Object failed with the following: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'In this case scenario, the SDK should be adjusting the signer with the skew and attempt the request again. (See. There is a client configuration parameter available enableClockSkewAdjustment which if set to true, adjusts clock skew after each http attempt and default to true. This might also be an issue with your retry strategy.
In the case of the 12MB file, the issue is caused by the Date header not being present in the response:
[2023-07-18 11:14:28.161] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [warning] [AWSClient] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2023-07-18 11:14:28.161] [org::apache::nifi::minifi::aws::utils::AWSSdkLogger] [debug] [AWSClient] Date header was not found in the response, can't attempt to detect clock skew
[2023-07-18 11:14:28.162] [org::apache::nifi::minifi::aws::s3::S3RequestSender] [error] PutS3Object failed with the following: 'Encountered network error when sending http request'due to Encountered network error when sending http request which causes the request to fail.
Now looking at the behavior using the CRT client, you are experiencing a similar behavior as the second case with a request timeout.
Based on the behavior above, it might be a network issue which would cause latency in the PutObjectRequest and would arise with larger file upload causing the behavior above.
There has been issue submission with similar behavior caused by this see.
Best regards,
Yasmine
lordgamez
commented
1 year ago Hi @yasminetalby ,
Thanks for investigating the issue. I checked and the enableClockSkewAdjustment was kept on the default true value, but I suppose the reason it did not do the clock skew adjusting because in the time diff check the TIME_DIFF_MAX and TIME_DIFF_MIN values are set to 4 minutes and the difference was only 1 second.
While testing it seems that I have found the root cause of the issue and it looks to be related to this issue. In the application code I had a default ServerSideEncryption::NOT_SET value set for the ServerSideEncryption and it was always set with this default value if no other value was specified by the user. After changing this to only set this value in the request when the configuration value is set to anything but "None" then the S3 upload works as intended on Windows:
inline constexpr std::array<std::pair<std::string_view, Aws::S3::Model::ServerSideEncryption>, 3> SERVER_SIDE_ENCRYPTION_MAP {{
{"None", Aws::S3::Model::ServerSideEncryption::NOT_SET},
{"AES256", Aws::S3::Model::ServerSideEncryption::AES256},
{"aws_kms", Aws::S3::Model::ServerSideEncryption::aws_kms},
}};
...
if (!put_object_params.server_side_encryption.empty() && put_object_params.server_side_encryption != "None") {
request.SetServerSideEncryption(minifi::utils::at(SERVER_SIDE_ENCRYPTION_MAP, put_object_params.server_side_encryption));
}So it seems that the issue mentioned in #1771 does not only happen with the CRT library but the WinHTTP library as well. When using a build on Linux with CURL for HTTP backend this issues does not appear.
After adding this fix I tested uploads with 5MB, 12MB, and 100MB sized files and all seem to work without a hiccup.
jmklix
commented
6 days ago I'm glad that you were able to get this working by setting the server side encryption. Please let us know if you have any other questions/potential bugs about this sdk.
github-actions[bot]
commented
6 days ago This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Describe the bug
I am building aws-sdk-cpp as a static library linked to my application with the following cmake flags:
-DBUILD_ONLY=s3 -DENABLE_TESTING=OFF -DBUILD_SHARED_LIBS=OFFI am using the SDK to upload files to S3 using the
PutObjectfunction ofS3Client. The application works fine on Linux, but there are some errors reported on Windows depending on the size of the upload.When the file to be uploaded is less than 10MB of size the upload fails with the error
SignatureDoesNotMatch(<bucket>is only a placeholder for the used bucket name):When the file to be uploaded is larger than 10MB it fails with a network error:
First tried with version 1.10.48, but also tried the latest version 1.11.118 with the same result. The issue was tested on AWS S3, but the same issue was reported when using S3 compatible storages: VAST S3 and MinIO S3
Suspected a firewall issue, but disabling the firewall had no effect.
Expected Behavior
PutObject call should succeed.
Current Behavior
PutObject failed.
Reproduction Steps
Aws::Auth::AWSCredentials = aws_credentialsprovider.getAWSCredentials(); auto client_config = Aws::Client::ClientConfiguration(); Aws::S3::Model::PutObjectRequest request; request.SetBucket("bucket"); request.SetKey("key"); auto stream = std::make_shared();
(*stream) << "Hello, World!";
request.SetBody(data_stream);
Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.PutObject(request);
Possible Solution
No response
Additional Information/Context
No response
AWS CPP SDK version used
1.11.118 and 1.10.48
Compiler and Version used
Visual Studio 2019, MSVC 14.29.30133
Operating System and version
Windows 10