Partial Support for S3 UploadPartCopy Functionality

ferristocrat commented 1 year ago

Description

We aim to implement a workaround to provide partial support for Amazon S3's UploadPartCopy functionality. This will primarily cater to customers attempting to move or copy whole files larger than 5GB, by identifying UploadPartCopy requests and triggering a server-side copy when the last range of an entire object comes in.

Problem/Pain Point

Our current architecture doesn't allow us to easily copy files by arbitrary ranges, leading to a lack of support for S3's UploadPartCopy. This has resulted in difficulties for key customers and integrations, especially those using S3 clients and libraries (such as boto3 for Python) that default to using UploadPartCopy for files over 5GB.

Why Now

This issue has been causing errors and dissatisfaction among our users. Given our current ability to perform server-side copy of entire objects, it is both a feasible and immediate solution to alleviate this problem and improve the reliability of our service.

Acceptance Criteria

Identification and handling of UploadPartCopy requests involving entire objects
Triggering of a server-side copy once the last range of an entire object comes in
Return of an error for UploadPartCopy requests involving arbitrary ranges

Out of Scope

Full compatibility with S3's UploadPartCopy
Copying files by arbitrary ranges

Success Metrics

Reduction in the number of errors experienced by customers using S3 clients for files over 5GB
Positive feedback from key customers and integrations about the partial UploadPartCopy support

Links

Relates to https://github.com/storj/roadmap/issues/40 (full compatibility)
Edge team milestone: https://github.com/storj/gateway-mt/milestone/26

ferristocrat commented 1 year ago

@amwolff - https://github.com/storj/gateway-mt/milestone/26 related milestone

iglesiasbrandon commented 1 year ago

@ferristocrat can you update the description and or title to better reflect what we are trying to achieve with this roadmap item?

ferristocrat commented 1 year ago

@amwolff - do we have enough info for a ROM, or do we need to complete https://github.com/storj/gateway-mt/issues/340 before we can determine ROM

amwolff commented 1 year ago

@ferristocrat It would be best to complete storj/gateway-mt#340 before the ROM on this one.

amwolff commented 12 months ago

Otherwise, I believe this is between M and L. More likely L than M, so that's what I picked in the dropdown here.

iglesiasbrandon commented 11 months ago

Moving this item back into the Up Next column because we have not started this effort and we had some other priorities come up.

novikov1337danil commented 6 months ago

This also causes an error when using storj as Amazon-S3 external storage in Nextcloud 28

nunesgh commented 6 months ago

This also causes an error when using storj as Amazon-S3 external storage in Nextcloud 28

Indeed, I have recently experienced the same issue.

storj / roadmap