Verify that the common case of UploadPartCopy is what AWS CLI does

[amwolff]

Goal

The objective of this task is to verify that AWS CLI uses the UploadPartCopy API to copy objects greater than 5 GB in size while maintaining the original multipart upload structure and not using arbitrary byte ranges or other methods. We can't support the latter, so this is to find out if we can conform to S3's internal limitation without reconfiguring all clients while using Storj's S3-compatible API.

Acceptance Criteria

• Successfully analyze and dump the requests issued by AWS CLI when copying objects greater than 5 GB.
• Verify that AWS CLI uses the UploadPartCopy API and maintains the original multipart upload structure.
• Ensure AWS CLI doesn't use arbitrary byte ranges or other methods to copy the object.
• Document the findings and any necessary recommendations or adjustments based on the analysis.

Links

• milestone
• Amazon S3 CopyObject API documentation
• Amazon S3 UploadPartCopy API documentation

[nergdron]

got a transparent proxy to run awscli through, connected to a personal bucket through the gateway. uploading a sufficiently large file to test copying with.

[nergdron]

ok, testing with minio. I've identified:

• CreateMultipartUpload:

  POST http://localhost:8912/test/d1.tar.zst?uploads HTTP/1.1

• UploadPart:

  PUT http://localhost:8912/test/d1.tar.zst?uploadId=NTYyODBjYWMtYTNlZC00NTA5LThhYzgtZGFhZjAwMTEyZTFhLjc1OTkwMmZiLTdiMTYtNGIxZS1iNWFhLWI5NWY1MGM2OTUzNQ&partNumber=7 HTTP/1.1

• CompleteMultipartUpload:

  POST http://localhost:8912/test/d1.tar.zst?uploadId=NTYyODBjYWMtYTNlZC00NTA5LThhYzgtZGFhZjAwMTEyZTFhLjc1OTkwMmZiLTdiMTYtNGIxZS1iNWFhLWI5NWY1MGM2OTUzNQ HTTP/1.1

that looks like the whole multipart upload workflow.

@amwolff is there anything else you're looking for here?

[amwolff]

@nergdron after you upload with multipart upload, you need to check that UploadPartCopy copies parts in the same manner that MPU uploaded parts (so while issuing a copy for a big enough object to trigger UploadPartCopy)

[nergdron]

alright, on an s3 copy, we get the following:

• initial read of source file:

  HEAD /test/d1.tar.zst HTTP/1.1

• CreateMultipartUpload:

  POST http://localhost:8912/test/d2.tar.zst?uploads HTTP/1.1

• UploadPart:

  PUT /test/d2.tar.zst?uploadId=OTcxZWE1M2EtM2ZjMy00NWE1LWJmYjktZWVkMWNlNzgyYTVmLjU2ZTZiZWVhLTdkYmItNDkyNC04MGQwLTg1OTRjMGNjMmQ1Ng&partNumber=10 HTTP/1.1

• CompleteMultipartUpload:

  POST /test/d2.tar.zst?uploadId=OTcxZWE1M2EtM2ZjMy00NWE1LWJmYjktZWVkMWNlNzgyYTVmLjU2ZTZiZWVhLTdkYmItNDkyNC04MGQwLTg1OTRjMGNjMmQ1Ng HTTP/1.1

so it appears to be using the same workflow as the original upload, but all the transfers are done server side, as the total packet capture for the copy is only 2.5MiB, instead of the ~12GiB for the original upload.

@amwolff, is this the behaviour you were looking for here? it's what I'd expect from reading the AWS docs.

[amwolff]

Do you have the log of requests/responses? It would be useful to see how many UploadPartCopies it's doing.

[nergdron]

This is the packet capture from doing a copy operation on a ~10G file inside of the same bucket. aws.pcap.gz

[amwolff]

from sprint planning: Paul reports that the CLI is doing what rclone does (#337) in the most common case.