search

rackspace / rack

A CLI for Rackspace (unmaintained)
Other
81 stars 57 forks source link

Add sync-dir function for Cloud Files #383

Open filler opened 8 years ago

filler commented 8 years ago

Hello --

First off, rack rules. Thanks for that. :smile_cat:

It would be helpful if there was a sync-dir function for Cloud Files. Much in the spirit of upload-dir, the idea is rack would only upload files to a named container which dont exist in the container but exist in the source of the sync-dir function. Also objects in the container which are not part of the source directory would be pruned.

There are many times when operators land files in a backup target directory on a source host, perform some life-cycle management (eg prune files older than n days) then want to shunt the remaining source files up to Cloud Files for safe-keeping (examples include db backups, log escrowing, etc).

Currently rack users have to empty a container and then call upload-dir in order to get a lifecycle-managed directory on the source into to the container. Thats rather inefficient for source bandwidth and the Files API.

It would be nice to optionally pass in an --prune option at sync-dir runtime to let the operator choose whether to remove files in the container which are not part of the source directory. Sometimes operators want to keep all the things in Files, other times operators do not and simply want a 1:1 relationship between files in the source directory and the container.

Duplicity is a tool which lots of operators use in order to incrementally sync directories into Cloud Files. Its an operational pain whereas rack isnt. :smile:

rclone has sync behavior like this by default. Given a fileset both on local disk and in a container, removing some of the files in the set on local disk and calling rclone sync removes them from the container they were previously synced in. It looks like it doesnt offer the operator the option to just shunt new local files on disk to the container and keeping the files in the container which were not on disk.

jrperritt commented 8 years ago

Thank you for filing this; it's something that was on my to-do list and got pushed back. I think I'll be able to work on it next week.

filler commented 8 years ago

That would be so :fire:. Thanks @jrperritt!

mbejda commented 7 years ago

Any update on this ?

jrperritt commented 7 years ago

Currently there are no plans to implement this.