Is there parity for `aws s3 sync`?

aws / aws-sdk-ruby

The official AWS SDK for Ruby.

https://aws.amazon.com/sdk-for-ruby/

Apache License 2.0

3.56k stars 1.22k forks source link

Is there parity for `aws s3 sync`? #824

Open csuhta opened 9 years ago

csuhta commented 9 years ago

The AWS CLI has the the amazing aws s3 sync command which I use frequently. It's super fast because in some cases it only copies the delta between the buckets.

There doesn't seem to be a similar way to invoke that API from this library. Is it here and I'm just missing it?

trevorrowe commented 9 years ago

The aws-sdk gem does not provide this functionality. As far as I know, none of the AWS SDKs, except the CLI have a filesystem to S3 sync function.

jufemaiz commented 7 years ago

davidcpell commented 6 years ago

As far as I know, none of the AWS SDKs, except the CLI have a filesystem to S3 sync function.

Hi, just wanted to stop by and add my :( to this. Sync functionality has been available in the PHP SDK since 2013. The Java SDK also has this/something similar.

shaunakv1 commented 6 years ago

Same here.. just stopping by again to add my :(

This is a really useful feature to have in sdk, so we don't have to install cli dependency on hosting, and our background jobs can run purely on gem dependency.

Adding this feature will make it whole lot easier to do PaaS deployments on system like Heroku.

pparamalingam commented 5 years ago

Bumping for 2019. Any chance of having this function added to the Ruby SDK?

george-viaud commented 4 years ago

Looking for SDK equivalent SDK method for aws s3 sync [SOURCE] s3://[BUCKET]/[KEYPREFIX] --exact-timestamps

Does this exist?

mullermp commented 4 years ago

Reopening as a feature request. Happy to take an PRs.

krysgian commented 4 years ago

Need the ability to run: aws s3 sync s3://accountOne/sourceBucket/etc s3://accountTwo/targetBucket/etc --delete --include="*"

Using the API, you can create a TransferManager object, but it assumes a single S3 AWS connection. An API change would hopefully accommodate this use case.

sudodevnull commented 3 years ago

Any update on this?

VitalyTarasiu commented 3 years ago

We are in 2021, any updates?

NemyaNation commented 3 years ago

^ @VitalyTarasiu I can second this. We are in 2021.

Still waiting for this feature to drop :(

dig12345 commented 3 years ago

ghost commented 3 years ago

Just do the loop yourself, the SDK only contains the actual API calls, not some utility that goes on top. It's not very hard. CS102, recurse through directories.

shaunakv1 commented 3 years ago

@pviet-cohere it's not "just a loop". May be read up on the documentation for the original utility?

https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

JVerma205 commented 2 years ago

Hi guys, I also am looking for SDK equivalent of CLI aws sync; I found this really helpful article that served my purpose. The code is easy to run as well. https://blog.bitsrc.io/a-practical-guide-to-building-a-node-js-service-on-an-aws-s3-bucket-aff19105ba83

Nevertheless, I do request aws-sdk to include the sync feature to standardize the algo.

midnight-wonderer commented 2 years ago

I know my comment will be unpopular; I braced for ☹️.
But, what's wrong with installing the AWS CLI? If you don't want it on the host, you can also install it inside a container.

I personally will choose neither SDK nor CLI for sync. Because, IMO, there is an economy behind software.
Old programmers are starting to die, literally; software, although still young on average, can live much longer than humans.
We are heading in the direction where there are not enough humans to maintain software; we are outnumbered. If not enough people are using it, it is potentially not viable.

You'll get better priority, support, and maintenance by switching to something more specific, like Rclone.
And posix-spawn isn't too much of a pain, is it?

Anyway, it seems the AWS team is still open for the PR; if you really want this, you should go for it.