search

Start9Labs / start-os

Open source Linux distro optimized for self-hosting
https://start9.com
MIT License
957 stars 89 forks source link

[feat]: versioned backups in TS sdk #2201

Open chrisguida opened 1 year ago

chrisguida commented 1 year ago

Would be great if we could use duplicity instead of rsync in TS, since duplicity creates incremental backups and doesn't overwrite all previous backups with the most recent one.

MattDHill commented 1 year ago

Rsync is also incremental, but yeah, no history. I think the way to handle this might be with Rsync but deciding client side whether to target an existing backup or a fresh backup. Meaning, instead of a creating a diff-based backup, package devs (and possibly users) could choose to create a fresh one (but still using Rsync). This could be accomplished simply by always creating then targeting a new directory that includes the date of the backup

chrisguida commented 1 year ago

I guess I should have said "versioned", not "incremental" xD. Renaming.

chrisguida commented 1 year ago

Wouldn't doing it the way you suggest remove the incrementailty? That is, if we backup to an entirely new directory, doesn't that duplicate all the data?

I guess that's what you're saying. Seems like we'd potentially be wasting a lot of space that way.

MattDHill commented 1 year ago

Correct, the choice would be between creating an incremental backup (overwriting the previous backup) or creating a totally fresh backup.

The third option would be creating incremental backups but retaining entire copies each time. This could also be done with rsync by cloning the dataset, saving the original, and then incrementing the clone.

Perhaps this is how Duplicity already works under the hood, but it would be nice to have a single standard across all services. When creating and recovering from backup, it should be uniform and expected for users

chrisguida commented 1 year ago

Right, this is what duplicity already does, that's what I mean by "versioned". You get snapshots, of every time you back up, rather than just a copy of whatever your datadir most recently looked like.

MattDHill commented 1 year ago

Presumably not without some additional effort. We've been using duplicity for over two years and people do not have access to old backups

MattDHill commented 1 month ago

SDK would still use rsync, but UI could choose to snapshot first