search

irods / irods_client_s3_api

C++ S3 API for iRODS
BSD 3-Clause "New" or "Revised" License
0 stars 5 forks source link

Support part upload with x-amz-copy-source #98

Open JustinKyleJames opened 8 months ago

JustinKyleJames commented 8 months ago

When a large file is copied, the aws client does a CreateMultipartUpload followed by PUT's which have the following:

Here is an example:

s3-api_1  | PUT
s3-api_1  | /alice-bucket/20Mfile2
s3-api_1  | partNumber=2&uploadId=19202868-e31f-4e1c-83cb-994f76a5b015
s3-api_1  | host:s3-api:8080 
s3-api_1  | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
s3-api_1  | x-amz-copy-source:alice-bucket/20Mfile
s3-api_1  | x-amz-copy-source-range:bytes=8388608-16777215

This is currently not supported. The current code will simply call the copy operation in iRODS after detecting the x-amz-copy-source header.

We will need an enhancement to support this.

As a workaround, multipart should be disabled for any copy operation. This is preferred anyway as it is far more efficient to call the iRODS copy operation than to read the bytes for each part, save a part file, and then stream it all back to iRODS on CompleteMultipartUpload.

As a workaround for the test cases, we will disable multipart for the copy tests.