Current code on master fails to handle large files on AWS S3 in 2 ways:
When performing multi-part uploading, the md5 given to upload_part is hexadecimal, but base64 is expected.
When calling S3.rename with a source object larger than 5GB, it fails due to limitations of copy_object.
While issue 1. is a straightforward fix. Solving issue 2. requires using multi-part copying, which (in my understanding) makes rename no longer atomic for large files. Considering rename is often used for its atomicity, I'm not sure whether this solution is best, and would be happy to get feedback.
Current code on
masterfails to handle large files on AWS S3 in 2 ways:upload_partis hexadecimal, but base64 is expected.S3.renamewith a source object larger than 5GB, it fails due to limitations ofcopy_object.While issue 1. is a straightforward fix. Solving issue 2. requires using multi-part copying, which (in my understanding) makes
renameno longer atomic for large files. Consideringrenameis often used for its atomicity, I'm not sure whether this solution is best, and would be happy to get feedback.