Scoring mechanism example

[Jip-Hop]

I tried out snapbtr and the simplicity is perfect for my use case. Every day I backup my files to a computer with a BTRFS filesystem. Now I want to keep daily snapshots of my files as well, so I can have versioned backups.

I keep my files in /media/user/backup and snapshots in /media/user/backup_snapshots. I'll be running this command as a daily cronjob for the root user: snapbtr -cs /media/user/backup /media/user/backup_snapshots --target-backups 100 --preserve-days 7

It works great and I like that I can easily browse the snapshot in the default file manager. I was even able to manually remove snapshots from the file manager, as standard user. Is it supposed to be like that? Because you mention "Root permissions are not required for creating snapshot" but I was also able to manually remove snapshots as non root user.

Could you provide an example of how the scoring mechanism works? With the above command (100 target backups and preserve 7 days). What would the snapshot folder look like in one month? After 6 months, one year and two years? I don't quite get the scoring mechanism.

I read through the code and noticed the oldest snapshot is never removed. Is this true? In my use case I don't need the oldest snapshot to stay around forever. So can I just manually remove it after a few months or so to free up disk space?

[yvolchkov]

I was even able to manually remove snapshots from the file manager, as standard user.

You should not be able to delete snapshot from a standard file manager, even as a root. Btrfs would not allow you to do this and it has nothing to do with snapbtr. Unless you have some fancy nautilus extension (do you have one actually?).

In other words, you can not delete snapshot by simple rm -rf test_snap you would need something like btrfs subvolume delete test_snap

Could you provide an example of how the scoring mechanism works?

To see examples of algorithm work, uncomment this lines:

        # Uncomment to visualize the algorithm
        # cand_str = ""
        # for (i,j) in remain:
        #     if j == mkill:
        #         j = Fore.RED + j + Fore.RESET
        #     elif j == mfrm or j == mto:
        #         j = Fore.GREEN + j + Fore.RESET
        #     cand_str += j + " "
        # print cand_str

and run:

./snapbtr --doctest

This will run a test simulation. Along with doctest (rightful) complains, you will see the algorithm visualization:

Every line is a step of the algorithm. A pair considered for deletion is colorised - green survived snapshot, red - killed.

You also could run simulation on your real folders:

./snapbtr -c --dry-run  --target-backups 3 /media/user/backup_snapshots 

With the above command (100 target backups and preserve 7 days).

Preserve 7 days means that snapbtr does not touch snapshots newer than 7 days at all. Like they do not exist.

I don't quite get the scoring mechanism.

In few words: directory names are converted to the time value and scaled exponentially. This means that time difference in 1 hour would make a bigger impact if the snapshot is from a month ago then from today.

And then distances between dirs are calculated. Algorithms finds a pair with the shortest distance (logarithmic) and kills one snapshot from this pair.

What would the snapshot folder look like in one month? After 6 months, one year and two years?

If you will be always running snapbtr as you wrote in the question you will have... 1) ... as many snapshots as you did within 7 days plus 2) ... one hundred of snapshots older then 7 days. The older snapshots - the bigger the gap between them

I read through the code and noticed the oldest snapshot is never removed. Is this true?

Yes, this is correct.

In my use case I don't need the oldest snapshot to stay around forever. So can I just manually remove it after a few months or so to free up disk space?

You can delete as many snapshots you want. Snapbtr does not keep status anywhere. It just reads the folder structure. You even can create snapshots manually, but you have to keep the same naming convention. However, I do not see any point in creating snapshots manually.

[Jip-Hop]

Thanks for replying. I'm using Lubuntu 16.04.3 with the default filemanager (pcmanfm) and no fancy addons. But maybe I did do something I shouldn't have. I did sudo chown user:user on both the /media/user/backup and /media/user/backup_snapshots folders. But I'm sure I didn't need to use btrfs subvolume delete test_snap to remove the snapshots when using the file manager. I can confirm that rm -rf test_snap is indeed not permitted.

I also just tried out snapbtrex. It's based on the same code as your project. Those snapshots are read only and I couldn't remove them using the file manager. I manually removed those snapshots with btrfs subvolume delete test_snap.

Thanks for your comments. I'll have a look at the visualisation. Perhaps you could have a look at snapbtrex. In your project I see the code related to remaining free space commented out. Seems like snapbtrex has this functionality. Also the --max-age argument addresses my desire to free up old snapshots.

[yvolchkov]

your file manager does not delete the snapshot. It just moves it in the trash. Command btrfs subvolume list <mnt_path> will reveal the truth.

➜  snapbtr git:(master) ✗ sudo btrfs subvolume list mnt
ID 257 gen 15 top level 5 path root
ID 258 gen 20 top level 5 path .Trash-364003917/expunged/3590358142
ID 259 gen 20 top level 5 path .Trash-364003917/expunged/1637015156
ID 260 gen 11 top level 5 path backups/20171109-184418
ID 261 gen 20 top level 5 path .Trash-364003917/expunged/3077674522
ID 262 gen 20 top level 5 path .Trash-364003917/expunged/2035007183

Those snapshots are read only and I couldn't remove them using the file manager

Actually, now when you mentioned it, I think snapbtr also has to create read-only snapshots. Because you could do a read-write snapshot of read-only snapshot at any moment..

Perhaps you could have a look at snapbtrex.

I did. I needed a simple backup solution, able to run from offlineimap hook as an unprivileged user. I considered all existing at that moment applications, none was able to do that. And everything required ridiculous amount of work for such simple thing or written in a language I do not like. I do not quite remember anymore what was wrong with snapbtrex, probably it became too sophisticated for my goals already.

I see the code related to remaining free space commented out

It is commented because it does not work. When you delete a snapshot, the free space amount is not updated immediately, and I do not know a how to synchronize it (btrfs subvolume sync does not work too). I guess snpbtrx also does not work with this option.

Also the --max-age argument addresses my desire to free up old snapshots.

I must agree, this is a useful option. I probably will implement it sometime soon.

[Jip-Hop]

Embarrassing! You're right about the trash folder. Glad to hear you find the --max-age option useful. With regards to the remaining free space I thought BTRFS quota groups (qgroups) were supposed to solve this issue. You could check the size of each snapshot with a single command and then calculate the freed space if you'd remove certain snapshots. But BTRFS is new territory to me so you probably considered this already.