Add rsync-like --update to s3 sync.

[coreyjewett]

It would be very powerful if sync copied the behavior of rsync's --update flag. This would allows predictable syncing from multiple directories without clobbering newer files. As it behaves right now, sync will update a file if the size OR timestamp differs.

In this example I'm creating a two folders, each with one file (test_file) having different sizes. I then create a bucket, sync the older, then the newer, then the older. As you can see from the output, the newer file is overwritten.

mkdir srcA srcB;
echo "older" > srcA/test_file;
sleep 2;
echo "this is newer" > srcB/test_file;

alias aws='aws --region=us-east-1';
ACL="--acl public-read";
bucket=$(uuidgen);

aws s3api create-bucket $ACL --bucket $bucket 
cd srcA; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
cd srcB; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
cd srcA; aws s3 sync $ACL . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file
aws s3 rm s3://$bucket/test/test_file
aws s3api delete-bucket --bucket $bucket 

=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> older
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> this is newer
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> older

With the --update flag in place like this:

cd srcA; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
cd srcB; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
cd srcA; aws s3 sync $ACL --update . s3://$bucket/test/; cd ..
curl http://$bucket.s3.amazonaws.com/test/test_file

I would expect the following output:

=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
=> upload: test_file to s3://a0dab2e5-2c66-49d8-971f-1bee05299ff3/test/test_file
# no output from the third command
=> this is newer

[vk6flab]

This is interesting. My version of aws s3 sync doesn't have an --update flag, so I didn't use it.

If I create the structure you show and "touch" the file, it updates as expected. That is, the third sync doesn't happen.

Having said that, there is no "sync" back to the local drive either, which I think is what you might mean when you talk about rsync like behaviour.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[kenorb]

Related:

• GH-1449: aws s3 sync --existing
• GH-2874: Add --no-overwrite option to aws s3 cp/mv
• Suggestion: Add --no-overwrite option to aws s3 cp/mv
• Suggestion: aws s3 sync --existing
• Suggestion: Add rsync-like --update to s3 sync

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[vk6flab]

Hallelujah!

On 7 April 2018 at 06:36, James Saryerwinnie notifications@github.com wrote:

Reopened #404 https://github.com/aws/aws-cli/issues/404.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws/aws-cli/issues/404#event-1561877304, or mute the thread https://github.com/notifications/unsubscribe-auth/ACPwwjSYTZnwoTZnf0suCp8KqNkhX0HAks5tl-3ZgaJpZM4BF02b .

-- Onno Benschop

()/)/)() ..ASCII for Onno.. |>>? ..EBCDIC for Onno.. --- -. -. --- ..Morse for Onno..

If you need to know: "What computer should I buy?" http://goo.gl/spsb66

ITmaze - ABN: 56 178 057 063 - ph: 04 1219 8888 - onno@itmaze.com.au

[acejam]

Bringing it back!

[yifeim]

+1 still finding this issue useful.

[mkieser]

this would be very useful

[kannanvr]

It is really useful for most of my use cases. When will this feature is available ?

[tim-finnigan]

Checking in as this issue hasn't be active in a long time. Regarding this original point:

As it behaves right now, sync will update a file if the size OR timestamp differs.

The --size-only parameter has since been added as the s3 sync documentation notes:

--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.

I'm not sure if that addresses all of the use cases here but I wanted to mention it in case it helps. If there's any use case that anyone wanted to expand on or clarify please let us know here.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[kenaniah]

still important

[MaheshB0ngani]

Really a needed feature. Still waiting for a solution to this.

[e8-HongJunLiu]

Really a needed feature. Still waiting for a solution to this.