Ideas around retention - deleting of old data

presslabs / z3

Backup your ZFS snapshots to S3.

https://www.presslabs.com/code/z3/

Other

375 stars 53 forks source link

Ideas around retention - deleting of old data #25

Open tgmedia-nz opened 6 years ago

tgmedia-nz commented 6 years ago

Hi guys, I hope I haven't missed reading it somewhere but I can't find how to keep data only for a certain amount of days, e.g.

keep 4 full backups keep 28 diff backups

As of now z3 will keep backing up without removing older backups, what do you guys use to keep data only for a specific time?

Cheers

pdobrescu commented 6 years ago

Hi @tgmedia-nz and thanks for using Z3. The way we're currently doing this is the following:

on the server where Z3 is running we have a script that does an automated ZFS cleanup of old snapshots (we keep them for 30 days)
on the S3 bucket we have an automatic expiry policy of 90 days and after that the backups are being removed automatically

As this is an open project and the use case above doesn't fit your needs, a PR with a retention option to be added to Z3 would be more than welcome!

CaerphillyMediaLtd commented 5 years ago

@pdobrescu If you remove them after 90 days aren't you deleting the files that contain the full system? Or are you doing a --full backup periodically?

bogdanpetrea commented 5 years ago

@CaerphillyMediaLtd: Yes, you are right. We do full backups periodically.