PutObject with ChecksumAlgorithm.SHA256 over unencrypted HTTP results in 400 error

skand888 commented 3 months ago

Describe the bug

When we specify ChecksumAlgorithm.SHA256, it's expected that the SDK will calculate the checksum and include it to the request.

However, the following code results in software.amazon.awssdk.services.s3.model.S3Exception: The provided 'x-amz-content-sha256' header does not match what was computed. (Service: S3, Status Code: 400, ...)

        var req = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(objectKey)
                .checksumAlgorithm(ChecksumAlgorithm.SHA256)
                .build();

        var resp = assertDoesNotThrow(
                () -> s3.putObject(req, RequestBody.fromFile(new File("test.txt"))));

It works as expected and completes sucsefully over HTTPS.

The entire request captured with Wireshark:

PUT /objectKey_putObjectWithSha256Algorithm HTTP/1.1
Host: fl-at-jv2-20240812-153626-664046833.s3.ap-southeast-1.amazonaws.com
amz-sdk-invocation-id: 57138d39-4375-a260-8e6b-99e10c058397
amz-sdk-request: attempt=1; max=4
Authorization: [removed]
Content-Type: text/plain
Expect: 100-continue
User-Agent: aws-sdk-java/2.26.29 Linux/5.15.153.1-microsoft-standard-WSL2 OpenJDK_64-Bit_Server_VM/21.0.4+7-LTS Java/21.0.4 vendor/Eclipse_Adoptium io/sync http/Apache cfg/retry-mode/legacy cfg/auth-source#stat ft/s3-transfer
x-amz-checksum-sha256: SMxWlcQBWBPkquqaEFUHaNZMNHb59jFjJ5p1DQueB7c=
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240812T153630Z
x-amz-sdk-checksum-algorithm: SHA256
Content-Length: 28
Connection: Keep-Alive

Actual file SHA256 is different from either SHA256 value:

$ sha256sum test.txt
8422b06904c61e5f05a2cd18add44479c6f4d8be2916adfbeb32eb69ab450b38  test.txt

$ sha256sum test.txt | awk '{print $1}' | xxd -r -p | base64
hCKwaQTGHl8Fos0YrdREecb02L4pFq376zLraatFCzg=

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is SHA256 of an empty zero-size file, SMxWlcQBWBPkquqaEFUHaNZMNHb59jFjJ5p1DQueB7c= is checksum of something else.

Expected Behavior

The code successfully uploads object via both HTTP and HTTPS.

Current Behavior

HTTPS: success HTTP: software.amazon.awssdk.services.s3.model.S3Exception: The provided 'x-amz-content-sha256' header does not match what was computed. (Service: S3, Status Code: 400, ...)

Other checksum algorithms work fine.

Reproduction Steps

Over HTTP endpoint URL:

        var req = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(objectKey)
                .checksumAlgorithm(ChecksumAlgorithm.SHA256)
                .build();

        var resp = s3.putObject(req, RequestBody.fromFile(new File("test.txt")));

Possible Solution

Properly calculate value for x-amz-checksum-sha256 even when x-amz-content-sha256 is used in signing process.

Additional Information/Context

No response

AWS Java SDK version used

2.26.29

JDK version used

21

Operating System and version

Linux

bhoradc commented 2 months ago

Hello @skand888,

Thank you for reporting the issue. The above S3 Exception - The SHA256 you specified did not match the calculated checksum. is reproducible over non-secure (http) S3 endpoint in Java SDK version 2.21.16 and above.

S3Client s3Client = S3Client.builder()
         .endpointOverride(URI.create("http://s3.us-east-1.amazonaws.com"))
         .build();

var req = PutObjectRequest.builder()
         .bucket("bucketname")
         .key("objectname")
         .checksumAlgorithm(ChecksumAlgorithm.SHA256)
         .build();

Path path = Paths.get("/Users/**/test.txt");
var resp = s3Client.putObject(req, path);

We will investigate further on this.

Regards, Chaitanya

bekzod-alibayev commented 1 month ago

@bhoradc has this issue been prioritized for any future releases? Having the same issue

bhoradc commented 1 month ago

Hi @bekzod-alibayev,

Thank you for reporting the issue. It's generally recommended to use HTTPS for communication with AWS services due to the security benefits it provides.

I am working with the team to see how we can reprioritize this issue. Meanwhile, could you please provide more details on your specific use case and reasons for preferring to use HTTP over HTTPS?

Regards, Chaitanya

bekzod-alibayev commented 1 month ago

Hi @bekzod-alibayev,

Thank you for reporting the issue. It's generally recommended to use HTTPS for communication with AWS services due to the security benefits it provides.

I am working with the team to see how we can reprioritize this issue. Meanwhile, could you please provide more details on your specific use case and reasons for preferring to use HTTP over HTTPS?

Regards, Chaitanya

We have Envoy sidecar container and all of the external traffic goes through it. In order to hit envoy container, it has to be http and envoy will do ssl for all external requests.

bhoradc commented 4 weeks ago

Hi @bekzod-alibayev,

Thank you for providing the use-case. Our team is working on prioritization of this issue and will get back to you when we have further updates on it.

Regards, Chaitanya

aws / aws-sdk-java-v2