seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

`cloudfiles rsync` Command #54

Open william-silversmith opened 3 years ago

william-silversmith commented 3 years ago

It's impossible to do a real rsync command with cloud storage b/c you can't replace pieces of files. However, you can check for whether a file is likely to be the same and not transfer it.

There's also the concept from zsync which performs all rsync computation on the client side and stores it in a metadata file on the dumb server. Would this be something we'd be interested in? It could help achieve rsync-like efficiencies across filesystems and assist with downloads (but not uploads) for object storage. Unlike rsync, it would require a pre-computed file being present which makes it less useful in general.

http://zsync.moria.org.uk/paper/ch02.html