search

aws / aws-sdk-java

The official AWS SDK for Java 1.x (In Maintenance Mode, End-of-Life on 12/31/2025). The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.13k stars 2.83k forks source link

TransferManager.upload doesn't retain `bucketKeyEnabled` setting if using multipart upload #3107

Closed ZhaoMJ closed 5 months ago

ZhaoMJ commented 7 months ago

Upcoming End-of-Support

Describe the bug

If multipart upload is used, files uploaded by the TransferManager don't have the correct object-level bucket key setting as specified in the PutObjectRequest. It turns out that the bucketKeyEnabled setting is not copied when configuring the InitiateMultipartUploadRequest: https://github.com/aws/aws-sdk-java/blob/34b4aa0668c1973252ef764d4768cf3ebfecc15c/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/UploadCallable.java#L422-L430

Expected Behavior

Objects uploaded by the TransferManager always have the correct object-level bucket key setting as specified in the PutObjectRequest.

Current Behavior

Objects uploaded by the TransferManager don't have the correct object-level bucket key setting as specified in the PutObjectRequest when multipart upload is used.

Reproduction Steps

  1. have a bucket with bucket-level bucket key disabled
  2. Create a TransferManager with a MultipartUploadThreshold of 8MB.
  3. Upload a file of 12MB using 
            PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, src)
                .withSSEAwsKeyManagementParams(sseAwsKeyManagementParams)
                .withBucketKeyEnabled(true);
  4. Verify that S3 bucket key is used for this file.
  5. Create another TransferManager with a MultipartUploadThreshold of 16MB.
  6. Upload the same file with the same PutObjectRequest using the new TransferManager.
  7. Verify that S3 bucket key is not enabled for this file.

Possible Solution

Copy bucketKeyEnabled from PutObjectRequest to InitiateMultipartUploadRequest, i.e. Add the following to UploadCallable.initiateMultipartUpload:

req.withBucketKeyEnabled(origReq.getBucketKeyEnabled())

Additional Information/Context

No response

AWS Java SDK version used

1.12.466

JDK version used

openjdk version "17.0.9" 2023-10-17 LTS OpenJDK Runtime Environment Zulu17.46+19-CA (build 17.0.9+8-LTS) OpenJDK 64-Bit Server VM Zulu17.46+19-CA (build 17.0.9+8-LTS, mixed mode, sharing)

Operating System and version

macOS 13.6.3

debora-ito commented 6 months ago

@ZhaoMJ thank you for raising this. Yes, it looks like bucketKeyEnabled is not being passed to InitiateMultipartUploadRequest. Marking as a bug.

The team's focus is in Java SDK v2 work, so this has low chance to be fixed before the start of Maintenance Mode later this year.

debora-ito commented 5 months ago

We don't have plans to fix before going into Maintenance Mode, so I'll go ahead and close this.

I've done a quick test using TransferManager v2, and this issue does not happen with S3 CRT client, the bucketKeyEnabled is correctly reflected in the initiateMultipartUpload request. So we recommend you migrate to Java SDK v2.

Reference:

github-actions[bot] commented 5 months ago

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.