search

AmesCornish / buttersink

Buttersink is like rsync for btrfs snapshots
GNU General Public License v3.0
195 stars 18 forks source link

S3 endpoint iterates through untargeted folders #63

Open vacri opened 6 years ago

vacri commented 6 years ago

Hi,

If I have an s3 target s3://mybackup/buttersink/, buttersink will iterate through all the files in s3://mybackup, ignoring the /buttersink/ restriction - so if I have a generic 'backup' bucket for various services, buttersink will needlessly go through all my backup files that live in other directories, which can take a while

Currently I've solved this by giving buttersink its own bucket, but this means I need to adjust user permissions to allow people to see that.

This is with pip-installed v0.6.9

AmesCornish commented 6 years ago

At the moment, I consider this a "feature". Buttersink will find needed diffs in other parts of your destination bucket and copy them over to the desired key. This saves all the expense of re-transferring a diff that's already in the destination bucket. Obviously there is a trade-off in that buttersink examines all the keys in the bucket. It would not be too hard to create a flag for this behavior. Thoughts?