Feature request - incremental backups with tar

[p1r473]

Hi Framps! have you looked into incremental backups with tar? It seems doable based on preliminary Google search.

https://linuxconfig.org/how-to-create-incremental-and-differential-backups-with-tar https://www.gnu.org/software/tar/manual/html_section/Incremental-Dumps.html#:~:text=Incremental%20backup%20is%20a%20special,restored%20when%20extracting%20the%20archive. https://www.geeksforgeeks.org/create-incremental-backup-in-linux-using-tar/ https://snapshooter.com/learn/linux/incremental-tar https://www.theurbanpenguin.com/incremental-backups-with-tar/ https://www.unixmen.com/performing-incremental-backups-using-tar/ https://www.golinuxcloud.com/tar-incremental-backup-example-linux/

[framps]

have you looked into incremental backups with tar?

I know it's possible but I didn't spent much time on this because rsync looks much easier to use.

But given the fact approx 1/3 of the raspiBackup users use tar I will check whether it's possible to add incremental tar support in raspiBackup.

[p1r473]

It would be absolutely great! Will put a donation bonus bounty on this 🤑

[framps]

I did some initial investigation:

1) There are new options required TAR_BACKUP_TYPE=normal|incremental|differential TAR_BACKUP_INCREMENTAL_PERIOD defines how much incremental or differential backups are created until a new full backup is created

2) Restore of these backups is much more complicated because TAR_BACKUP_INCREMENTAL_PERIOD+1 tar restore runs have to be executed

3) Question is how to organize the full and incremental/differential backups. All backups created by raspiBackup can and should be able to be restored manually. Therefore all incremental backups should be collected in a subdirectory of the full backup. Current backup recycle strategy will not be able to handle this subdirectory structure.

4) Every TAR_BACKUP_INCREMENTAL_PERIOD a full backup will be created. As of now every backup usually takes the same amount of time and the time is predictable. Now there are backups which are fast and backups with are slow.

5) Backup meta information is encoded in the backup directory name. The encoding scheme has to be updated in order to include incremental and delta backups and will require major recycle algorithm changes

[framps]

I just created a poll to get an idea whether it's worth to spend the huge effort required to support an incremental tar backup.

[framps]

@p1r473 I created a small POC for the inc and diff tar backup. The inc backup will be very time consuming because the backup rotation strategy will become quite complicated and the restore logic will also be affected.

In contrast the diff backup will not complicate the rotation strategy and restore because every diff backup is a full backup.

Typical usecases with diff backup I see will be:

1) Daily backups are created. Every 7 days a new full backup is created and all following backups are diff backups. TAR_BACKUP_DIFFERENTIAL_PERIOD=7 2) Weekly backups are created. Every 4 backups a new full backup will be created. TAR_BACKUP_DIFFERENTIAL_PERIOD=4

In https://github.com/framps/raspiBackup/issues/625#issuecomment-1452385685 I thought it's useful to have both backup methods supported by raspiBackup. But that's as I wrote too heavy to implement.

Let me know whether you agree the diff backup is also useful to have in raspiBackup. It's not a diff backup in a pure sense but I think it's better than to create all the time a full tar backup.

[p1r473]

I agree that diff and incremental are both great features to have. Some people will use full backups, some people diffs, some people incremental, and that's what makes a great product 😁 options!

[framps]

Ok. I'll add diff backups in raspiBackup. inc backups are out of scope. I'll create a new feature request for diff backups and put this request for inc backups in the backlog.