search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.06k stars 573 forks source link

@aws-sdk/lib-storage: multipart copy #3550

Open timkye-blueocean-ai opened 2 years ago

timkye-blueocean-ai commented 2 years ago

This is a copy of this previous issue

Is your feature request related to a problem? Please describe.

I'm trying to copy arbitrary-sized objects between buckets.

The objects could be between 0KB and 5TB in size.

Describe the solution you'd like

I would like the new Upload class in @aws-sdk/lib-storage to support copy operations.

Describe alternatives you've considered

AFAIK the only alternatives with v3 are:

  1. To manually write a lot of code to detect if the file is < 5GB and use s3.copyObject if so, or enter into a mutlipart copy operation if not, using s3.uploadPartCopy plus all of the other orchestration around it... which is a lot, when you consider error handling, etc.
  2. To use the new Upload class, but with s3.getObject().createReadStream() as a Body. This would obviously involve all data having to go through my service, so will perform poorly.
Jordan-Hall commented 1 year ago

Is the solution here to have a new CopyUpload class or do you allow two types of Params not just PutObjectCommandInput then internally determine the correct command to use? I'm looking at this now until a proper solution is added and I would prefer to be closer to the actual solution

mappingvermont commented 1 year ago

Bumping this 🙏

klausbadelt commented 7 months ago

@Jordan-Hall My preference would be extending the existing Upload class. Instead of params.Body we'd provide CopySource, just like in CopyObjectCommand.

I could see a separate Copy class too though, if simpler.

RanVaknin commented 6 months ago

Hi there,

This is a legitimate feature request ,I have assigned it standard priority.

Thanks, Ran~