The objective of this task is to verify that AWS SDK uses the UploadPartCopy API to copy objects greater than 5 GB in size while maintaining the original multipart upload structure and not using arbitrary byte ranges or other methods. We can't support the latter, so this is to find out if we can conform to S3's internal limitation without reconfiguring all clients while using Storj's S3-compatible API.
Acceptance Criteria
Successfully analyze and dump the requests issued by AWS SDK when copying objects greater than 5 GB.
Verify that AWS SDK uses the UploadPartCopy API and maintains the original multipart upload structure.
Ensure AWS SDK doesn't use arbitrary byte ranges or other methods to copy the object.
Document the findings and any necessary recommendations or adjustments based on the analysis.
Experiment with changing the default threshold and the threshold itself for triggering the multipart mechanism (applicable to both uploads and copies) throughout the experiment to see how it affects what SDK is doing. Ideally, you want to use some abstraction over UploadPartCopy existing in the SDK, if any, as that should be what developers prefer.
Goal
The objective of this task is to verify that AWS SDK uses the UploadPartCopy API to copy objects greater than 5 GB in size while maintaining the original multipart upload structure and not using arbitrary byte ranges or other methods. We can't support the latter, so this is to find out if we can conform to S3's internal limitation without reconfiguring all clients while using Storj's S3-compatible API.
Acceptance Criteria
Experiment with changing the default threshold and the threshold itself for triggering the multipart mechanism (applicable to both uploads and copies) throughout the experiment to see how it affects what SDK is doing. Ideally, you want to use some abstraction over UploadPartCopy existing in the SDK, if any, as that should be what developers prefer.
Links