Option to backup modified or deleted files

[br1985]

In backup-like scenarios something similar to 'rsync -b' would be extreamly useful.

This options tells rsync to rename (appending specified suffix) and optionally move (to a specified directory) any file that is going to be overridden or deleted.

Without this simple automatic rsync-based backup jobs are just too risky. If something bad happens with the source (e.g. your files were accidentaly truncated) all changes will simply propagate to the target effectively destroying you backup.

The '-b' option prevents this and greatly simplifies your jobs.

[mfschwartz]

Hi - would it work for you to enable versioning on the destination bucket? With versioning enabled, if something goes wrong during the rsync any overwritten or deleted objects will end up as versions you can recover. If this won't work for you please let us know why.

[br1985]

Hi! This should work just fine for us. Thank you for this idea! I was aware of such option.

[komasoftware]

Hi @mfschwartz

From the docs : "The gsutil rsync command ignores versioning, synchronizing only the live object versions in versioned buckets"

I am doing backups with gsutil rsync and I checked one of the modified files and there are no versions created by gsutil rsync so this solution won't work. Real bummer !

Koen

[mfschwartz]

Hi Koen,

I'm not clear on what the problem is that you're reporting. I believe gsutil's synchronizing only the live objects in versioned buckets is what you need to backup your data.

About your second point: when you talk about "modified files" are you saying gsutil updated an object in the destination bucket (based on the source object having changed) and an archived version of that object didn't get created even though versioning is enabled on that bucket?

If not, please provide more detail about what you did and what didn't work as expected.

Thanks.

[komasoftware]

My mistake... rsync does indeed create versions. Can you explain then what this sentence in the docs means : "The gsutil rsync command ignores versioning, synchronizing only the live object versions in versioned buckets" ? My understanding was that this would not create versions but update the live object only.

[mfschwartz]

That sentence means rsync considers only the current object generations in the source and destination bucket when deciding what to copy / delete. If versioning is enabled in the destination bucket then overwriting or deleting objects will end up creating versions, but gsutil rsync doesn't try to make the archived generations match in the source and destination buckets.

[komasoftware]

Ok thx a lot... replacing "objects" with "source objects" in the docs would make it easier to understand IMO. Again, thank you for the explanation.

[mfschwartz]

I edited my answer because it wasn't quite clear/accurate before.

[komasoftware]

OK, I understand. So it is not only "source" objects and the docs are correct. And I'd say it is expected behaviour. Short sentence in the docs.. alot to talk about ;) Thx.

[mfschwartz]

Thanks, I updated the gsutil rsync documentation so this will be more clear. The updated doc will be included in the 4.14 release.