search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.62k stars 230 forks source link

EntityTooLarge: fail to cp file with 6.3GB from one bucket to another bucket #29

Open tony710508 opened 4 years ago

tony710508 commented 4 years ago

fail to cp file with 6.3GB from one bucket to another bucket

EntityTooLarge: Your proposed upload exceeds the maximum allowed object size

JoshuaEdwards1991 commented 4 years ago

I have also just encountered this issue, has something changed? Note: this is between s3 buckets and not from EBS to S3

igungor commented 4 years ago

Thanks for the report. We're using the official AWS SDK. Copying objects (from S3 to S3) larger than 5GB requires multipart uploading which we currently don't leverage.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

You create a copy of your object up to 5 GB in size in a single atomic operation using this API. However, for copying an object greater than 5 GB, you must use the multipart upload Upload Part - Copy API.

We're going to use the multipart Copy API to support large file transfers between S3 buckets. Please note that uploading a large file to S3 works as expected.

igungor commented 4 years ago

https://github.com/aws/aws-sdk-go/pull/2653 PR is required to address this issue.

AravindhStanley commented 3 years ago

Just checking if there is any news on this issues. I could make use of this feature in my workflow. Not sure that PR @igungor mentioned is merged/ what's stopping that merge.

rosibaj commented 2 years ago

Is there any update on the status of this issue?

batic commented 2 years ago

Similar problem is with sync when file is larger than 5G.

az-z commented 1 year ago

case: source - has objects dest - is empty

s3cmd sync source/* dest/

"sync" fails silently on the first run after copying some objects. "sync" will fail on the second run and report the offending object

az-z commented 1 year ago

actually, bring us to the question - how do we exclude files from cp/sync ?

tooptoop4 commented 1 year ago

some other go libs support this ie https://github.com/rclone/rclone/commit/7b1274e29ac512319b5555090a78cd15cf277dd6 or https://github.com/drakkan/sftpgo/commit/6f8b71b89f81787fa50bd04cbde20c2e8c7a08c7#diff-53095c9aca755cd4fe0455d2549f3668df445ac69801f752b2371bcd8a4c5efaR810

matan129 commented 1 year ago

Any updates?

tweep commented 1 year ago

Ran into this today - any updates on this or is there a work-around ?

tooptoop4 commented 1 month ago

"InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 status code: 400" this is the error in v2.2.2