Figure out a backup strategy for small, changing files

[ctengel]

• In the past we used rsync
• our new solution is optimized for WORM
• let's assume all large media is being handled directly via obj store
• need some other software, tar up daily incrementals? to create a large object
• needs to be recoverable!

[ctengel]

• tar seems to have a mechanism simply by relying on a single file (which doesn't need to be backed up as only needed when creating subsequent incremental backups; not needed for restoral) https://www.gnu.org/software/tar/manual/html_section/Incremental-Dumps.html
• the other obvious choice would be btrfs snapshots... similar to how we did zfs snapshots and zfs send streams
• regardless we want something that can easily be run weekly/nightly incrementally and monthly/quarterly/annually in full

[ctengel]

Let's plan to write a quick script that leverages tar...

Simple directory with one file per level 0 filesystem backup... filename contains info like

• host
• device
• mountpoint
• date of level 0 backup
• tar list fine or metadata file

in the metadata file are entries

• tar list file
• date of backup
• level of backup
• object index ID

1. on local machine, look for recent incremental file
   • create new if too old
2. check for corresponding OI objects
3. setup copies of tar list file if needed
4. run backup
5. upload/stream to OI
6. finalize log/metadata file

if we switch to btrfs do that, but short to medium term likely multiple systems that WONT support it, even if medium term some do switch

8. upload result to oi with appropriate tagging

[ctengel]

Note that the script looking for stuff could use #27 for base use... but no need to jump to #20 ... mediacrawler for backups? i.e. should there be a microservice vs using local files? not for MVP...

[ctengel]

actually the less in the flatfiles the better - we won't have those at restore time! a seperate backup service should not be needed if an appropriate tagging is in objectindex. basically some way to search for available backups of a given host

do we need to have a way to stream these tarfiles to OI?

[ctengel]

Let's say this depends on #11 - this script would simply call that most likely