Integration with snapper

rogueai commented 2 years ago

Hi all,

First of all, apologies if this was asked before, but I'm trying to figure out if btrbk can fit my workflow. A little background: I'm using snapper to manage my local snapshots, as per default configuration I have all my snapshots located in /.snapshots. I understand btrbk can pretty much cover all snapper use cases, but for the time being I think I'll stick with it as my config is somewhat tied to it (especially pacman hooks and automatic restoring through refind-btrfs).

So my question is: can btrbk be used just for backing up the snapshots to an external location? I'm currently backing everything up manually via btrfs send/receive, but that's becoming a bit of a chore.

Thanks!

camoz commented 2 years ago

Cannot answer your original question, but here are some alternatives: https://wiki.archlinux.org/title/Btrfs#Incremental_backup_to_external_drive

Zulgrib commented 2 years ago

Hello camoz,

After evaluating all links in archlinux wiki, btrbk seems to be the way to go, but it wouldn't hurt to have snapper compatibility.

Can this feature request be evaluated @digint ? Use would be for a file server to pull snapper's snapshots from a host, or a host to push snapper's snapshot to a server.

digint commented 2 years ago

Unfortunately, btrbk relies on its own naming scheme, and most probably cannot backup snapshots from snapper because of that. I'm working on a rewrite of btrbk archive which might be able to work for all subvolume namings. Not sure though if this will be possible, I'll try.

Sadly I'm very busy lately, don't expect this too soon...

camoz commented 2 years ago

I'm not sure this is a good idea. Every feature comes with a maintenance cost, and there already exist dedicated tools for backing up snapper snapshots to external locations (which are listed in the archwiki link I posted above), like https://github.com/doudou/snapsync and https://github.com/qubidt/snap-sync.

digint commented 2 years ago

I have implementing basic "timeshift" support for btrbk cp (see #480). Can someone here please tell me how snapper creates its snapshots? After a first look through the docs it seems like it keeps metadata (like timestamps) in dedicated files?

rogueai commented 2 years ago

I have implementing basic "timeshift" support for btrbk cp (see #480). Can someone here please tell me how snapper creates its snapshots? After a first look through the docs it seems like it keeps metadata (like timestamps) in dedicated files?

Hi, first of all thanks for looking into this.

Snapper keeps all snapshots in folders under your snapshots' target directory, e.g.: /<snapshots_target>/<snapshot_id>/. The folder structure looks something like this:

/.snapshots
├── 1
│   ├── info.xml
│   └── snapshot
├── 100
│   ├── info.xml
│   └── snapshot
├── 101
│   ├── filelist-100.txt.gz
│   ├── info.xml
│   └── snapshot
├── 200
│   ├── info.xml
│   └── snapshot

the snapshot with id 1 is always there and represent the initial parent snapshot
info.xml: snapper's metadata
filelist-<snapshot_id>.txt.gz: gets created with pre/post snapshots, which are snapshots related to each other and always go in pair. In this example 100 is a pre-snapshot and 101 its paired post-snapshot. Note that sometimes it's not compressed, and looks like a regular txt file
snapshot/: the actual btrfs snapshot

I seem to have a lot of folders with info.xml, but no snapshot/ folder inside, I suppose they're old snapshot that didn't get properly cleaned up in maintenance operations. They can be ignored as far as I can tell, but something you might want to keep an eye on.

To give a bit of context, I'm currently using a script I mashed together to send snapshots remotely over ssh, the basic gist of it is:

create a the target folder <snapshot_id>
btrfs send/receive the actual snapshot to <snapshot_id/snapshot
scp or rsync snapper's metadata files

Let me know if you need more info, I'll be glad to help.

zilexa commented 2 years ago

Honestly, Snapper is created based on such a different philosophy than btrbk (and Timeshift for that matter) I would like to humbly suggest to skip working on improving Snapper support.

Not only does Snapper use separate metadata files, it also always creates snapshots within the source subvolume. This is a pain if you actually need to rollback to an older subvolume. You would be forced to move all snapshots, mess with fstab etc. It just creates unnecessary complexity. I stopped using it.

Btrbk can already do everything that snapper can do. For a few things it's simply easier/more user friendly to use a UI, but then Timeshift is great. And btrbk already supports copying those snapshots to a target.

rogueai commented 2 years ago

Btrbk can already do everything that snapper can do. For a few things it's simply easier/more user friendly to use a UI, but then Timeshift is great. And btrbk already supports copying those snapshots to a target.

I see your point, and I might actually start investigating a transition to btrbk and be done with snapper altogether. The only things that really stopped me from moving are:

pacman hooks: I might actually setup my own hook for pre/post pacman transactions, it shouldn't be too much of a problem
refind stanzas integration: I'm using refind-btrfs to create dynamic stanzas based on snapshots to easily start up a snapshot from refind. The project seems to be heavily tailored around snapper, I might actually investigate if it's possible to configure it to use btrbk snapshots, or move to a different solution altogether.

Massimo-B commented 2 years ago

I have scenarios where I combined snapper and btrbk... First about the story: When I decided to use btrbk or snapper I've seen snapper is quite powerful but does not support sending snapshots to other devices, so the decision was clear for going on with btrbk.

But I have some OpenSuse machines that are well integrated with snapper, like the zypper package manager does snapshots for every update. This framework I just kept. So I let snapper do the root snapshots while btrbk does the other snapshots and send all snapshots to the backup device. Note that the default subvolume is set to "@/.snapshots/1/snapshot" which is the current working root fs and some snapper or OpenSuse default. As snapshot_dir I took the same target that snapper is using: @/.snapshots/root So snapper is creating snapshots there and btrbk as well. Now that I see from your evaluation that snapper is creating it's own metadata, it may be that snapper would not be able to use the snapshots created by btrbk. To be honest I never switched back to some other root snapshot via grub or default subvolume. I just need the automatic snapshot creation by zypper and do the rest with btrbk.

And here is my setup:

# grep btrfs /etc/fstab  |grep -v "^#"
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /                   btrfs   defaults,noatime,nodiratime,compress-force=zstd:3       0 0
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /var/cache          btrfs   noatime,nodiratime,compress-force=zstd:3,subvol=/@/var/cache    0 0
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /tmp                btrfs   noatime,nodiratime,compress-force=zstd:3,subvol=/@/tmp          0 0
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /home               btrfs   noatime,nodiratime,compress-force=zstd:3,subvol=/@/home         0 0
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /.snapshots         btrfs   noatime,nodiratime,compress-force=zstd:3,subvol=/@/.snapshots   0 0
UUID=f9d055cc-2507-42b7-4ce3-9ef6a2c754c0   /mnt/btrfs-top-lvl  btrfs   noatime,nodiratime,compress-force=zstd:3,subvol=/           0 0
LABEL=mobile_data               /mnt/usb/mobiledata btrfs   noauto,x-systemd.automount,x-systemd.idle-timeout=300,noatime,nodiratime,compress-force=zstd,space_cache=v2,async 0 0

# grep -v "^#" /etc/btrbk/btrbk.conf
archive_preserve        12m 10y
archive_preserve_min    latest
incremental             yes
lockfile                /var/log/btrbk.lock
stream_buffer           5%
timestamp_format        long-iso
transaction_log         /var/log/btrbk.log

volume /mnt/btrfs-top-lvl/
    snapshot_create  onchange

    subvolume @/.snapshots/1/snapshot
        group root
        snapshot_dir            @/.snapshots/root
        snapshot_preserve_min   5d
        snapshot_preserve       2d 1w 1m
        target send-receive     /mnt/usb/mobiledata/snapshots/lintop/root
            target_preserve_min     latest
            target_preserve         1d 4w 2m 1y

    subvolume @/home
        group home
        snapshot_dir            @/.snapshots/home
        snapshot_preserve_min   3d
        snapshot_preserve       4d 2w 1m
        target send-receive     /mnt/usb/mobiledata/snapshots/lintop/home
            target_preserve_min     3d
            target_preserve         6d 5w 4m 3y

n8henrie commented 1 year ago

@Massimo-B can you give any additional detail about how you got this working? Snapper creates read-only snapshots, so when I try to use e.g. subvolume /.snapshots/1/snapshot I am thwarted by !!! Aborted: subvolume is readonly.

Massimo-B commented 9 months ago

Part of my almost complete config: https://github.com/digint/btrbk/issues/560#issuecomment-1737717208

digint / btrbk

Integration with snapper #456