Verify that the common case of UploadPartCopy is what rclone does

amwolff commented 1 year ago

Goal

The objective of this task is to verify that rclone uses the UploadPartCopy API to copy objects greater than 5 GB in size while maintaining the original multipart upload structure and not using arbitrary byte ranges or other methods. We can't support the latter, so this is to find out if we can conform to S3's internal limitation without reconfiguring all clients while using Storj's S3-compatible API.

Acceptance Criteria

Successfully analyze and dump the requests issued by rclone when copying objects greater than 5 GB.
Verify that rclone uses the UploadPartCopy API and maintains the original multipart upload structure.
Ensure rclone doesn't use arbitrary byte ranges or other methods to copy the object.
Document the findings and any necessary recommendations or adjustments based on the analysis.

Links

pwilloughby commented 1 year ago

Rclone uses the UploadPartCopy API. Multipart uploads use part sizes of 5Mi by default and uploadPartCopy uses part sizes of 4.656Gi by default. The byte ranges are calculated here using part sizes from the --s3-copy-cutoff config option and not the original upload structure. Based on my reading of https://github.com/storj/storj/blob/main/docs/blueprints/byte-range-multipart-copy/byte-range-multipart-copy.md#Restrictions-compared-to-S3-API we should be able to support this with the current proposal.

pwilloughby commented 1 year ago

Here's a trace of the calls rclone makes

$ rclone --version
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18.1
$ rclone copy --help minio:ham/6gb_test minio:ham/copy_6gb

./mc --config-dir . admin trace --all localhost
21:34:08.699 [200 OK] s3.HeadObject localhost:8000/ham/6gb_test 127.0.0.1         738µs       ↑ 77 B ↓ 417 B
21:34:08.701 [404 Not Found] s3.HeadObject localhost:8000/ham/copy_6gb 127.0.0.1         153µs       ↑ 77 B ↓ 200 B
21:34:08.702 [200 OK] s3.HeadObject localhost:8000/ham/6gb_test 127.0.0.1         1.102ms      ↑ 77 B ↓ 417 B
21:34:08.703 [404 Not Found] s3.HeadObject localhost:8000/ham/copy_6gb/6gb_test 127.0.0.1         116µs       ↑ 77 B ↓ 200 B
21:34:08.705 [409 Conflict] s3.PutBucket localhost:8000/ham 127.0.0.1         948µs       ↑ 103 B ↓ 629 B
21:34:08.706 [200 OK] s3.HeadObject localhost:8000/ham/6gb_test 127.0.0.1         678µs       ↑ 77 B ↓ 417 B
21:34:08.708 [200 OK] s3.NewMultipartUpload localhost:8000/ham/copy_6gb/6gb_test?uploads=  127.0.0.1         481µs       ↑ 141 B ↓ 480 B
21:34:08.709 [200 OK] s3.CopyObjectPart localhost:8000/ham/copy_6gb/6gb_test?partNumber=1&uploadId=f3c20423-2ffc-4248-b108-9ba9ed8dabc0  127.0.0.1         24.406181s   ↑ 135 B ↓ 453 B
21:34:33.123 [200 OK] s3.CopyObjectPart localhost:8000/ham/copy_6gb/6gb_test?partNumber=2&uploadId=f3c20423-2ffc-4248-b108-9ba9ed8dabc0  127.0.0.1         10.165858s   ↑ 135 B ↓ 453 B
21:34:43.298 [200 OK] s3.CompleteMultipartUpload localhost:8000/ham/copy_6gb/6gb_test?uploadId=f3c20423-2ffc-4248-b108-9ba9ed8dabc0  127.0.0.1         8.01033s     ↑ 380 B ↓ 633 B
21:34:51.312 [200 OK] s3.HeadObject localhost:8000/ham/copy_6gb/6gb_test 127.0.0.1         2.242ms      ↑ 77 B ↓ 409 B

storj / edge