digint / btrbk

Tool for creating snapshots and remote backups of btrfs subvolumes
https://digint.ch/btrbk/
GNU General Public License v3.0
1.71k stars 123 forks source link

Only transfer snapshots #560

Open Massimo-B opened 1 year ago

Massimo-B commented 1 year ago

How can I only transfer snapshots without creating new ones? I could use btrbk archive for that, but I try to integrate it into my central btrbk.conf.

The scenario is like this:

I have several machines. Each machine has a local data store. Then I have a mobile data store that I carry with from machine to machine. The mobile data store is the central store which will have all snapshots from all machines. Each machine should send its snapshots to its own local datastore and also to the mobile datastore, if currently available. Each machine will also try to reach other machines to copy snapshots from them. Now I added the section "Reverse Backups" in order to copy snapshots from mobile datastore back to the local datastore for backup duplication.

For instance. the attached example btrbk.conf is from desktop_one. It will send its snapshots to local and also mobile. It also tries to reach other machines like laptop_one, server_one and laptop_two, if available. It will also send their snapshots to the mobile datastore, but also to some archive dir on its local datastore. Because some machines don't see each other or only see each other online only infrequently, the machine should try to find snapshots from the central mobile datastore and copy them back to the local datastore.

I tried to follow the example https://github.com/digint/btrbk#example-multiple-btrbk-instances because this only sends without changing the source. But it seems I always need to reference the original subvolume of the snapshot. That doesn't work in my situation. Being on desktop_one it should copy snapshot of desktop_two back from mobile to local, I have no access the the original subvolume at that time.

Is btrbk archive the only way to do that?

btrbk-public.conf.txt

btw. laptop_two is an OpenSuse machine, btrbk is adapting to the co-existing Snapper scheme there.

Massimo-B commented 1 year ago
archive_preserve        2w
archive_preserve_min    latest
incremental             yes
lockfile                /var/log/btrbk/btrbk.lock
preserve_day_of_week    monday
#send_compressed_data    no
#send_protocol           1
snapshot_create         onchange
snapshot_preserve       14d 5w 2m
snapshot_preserve_min   1w
ssh_identity            /root/.ssh/id_ed25519
stream_buffer           5%
stream_buffer_remote    5%
stream_compress         lz4
stream_compress_adapt   no
stream_compress_level   default
stream_compress_threads default
target_preserve_min     1w
timestamp_format        long-iso
transaction_log         /var/log/btrbk/btrbk.log

###################################################################################
volume /mnt/btrfs-top-lvl/
    group               all,cron,local
    #-------------------------------------
    subvolume volumes/root
        group                   root
        snapshot_dir            snapshots/root
        target send-receive     /mnt/local/data/snapshots/root
            group                   localdata
            target_preserve         5d 5w 3m
        target send-receive     /mnt/usb/mobiledata/snapshots/desktop_one/root
            group                   mobiledata
            target_preserve         30d 8w 12m 3y
    #-------------------------------------
    subvolume volumes/home
        group                   home
        snapshot_dir            snapshots/home
        target send-receive     /mnt/local/data/snapshots/home
            group                   localdata
            target_preserve         14d 8w 6m 1y
        target send-receive     /mnt/usb/mobiledata/snapshots/desktop_one/home
            group                   mobiledata
            target_preserve         30d 24w 12m 8y
    #-------------------------------------
    subvolume volumes/data
        group                   data
        snapshot_dir            snapshots/data
        target send-receive     /mnt/local/data/snapshots/data
            group                   localdata
            target_preserve         3d 4w 6m
        target send-receive     /mnt/usb/mobiledata/snapshots/desktop_one/data
            group                   mobiledata
            target_preserve         6d 8w 12m 2y
    #-------------------------------------
    subvolume volumes/vm
        group                   vm
        snapshot_dir            snapshots/vm
        target send-receive     /mnt/local/data/snapshots/vm
            group                   localdata
            target_preserve         1m
        target send-receive     /mnt/usb/mobiledata/snapshots/desktop_one/vm
            group                   mobiledata
            target_preserve         6m 1y
    #-------------------------------------
    subvolume volumes/workspace
        group                   workspace
        snapshot_dir            snapshots/workspace
        target send-receive     /mnt/local/data/snapshots/workspace
            group                   localdata
            target_preserve         14d 8w 6m 1y
        target send-receive     /mnt/usb/mobiledata/snapshots/desktop_one/workspace
            group                   mobiledata
            target_preserve         30d 24w 12m 8y

###################################################################################
volume ssh://laptop_one/mnt/btrfs-top-lvl/
    send_compressed_data    yes
    send_protocol           2
    group                   all,cron,remote,ssh,laptop_one
    #-------------------------------------
    subvolume volumes/root
        group                   root
        snapshot_dir            snapshots/root
        target send-receive     /mnt/local/data/archive/laptop_one/root
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_one/root
            group                   mobiledata
            target_preserve         30d 8w 12m 3y
    #-------------------------------------
    subvolume volumes/home
        group                   home
        snapshot_dir            snapshots/home
        target send-receive     /mnt/local/data/archive/laptop_one/home
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_one/home
            group                   mobiledata
            target_preserve         30d 24w 12m 8y
    #-------------------------------------
    subvolume volumes/data
        group                   data
        snapshot_dir            snapshots/data
        target send-receive     /mnt/local/data/archive/laptop_one/data
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_one/data
            group                   mobiledata
            target_preserve         6d 8w 12m 2y
    #-------------------------------------
    subvolume volumes/vm
        group                   vm
        snapshot_dir            snapshots/vm
        target send-receive     /mnt/local/data/archive/laptop_one/vm
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_one/vm
            group                   mobiledata
            target_preserve         6m 1y
    #-------------------------------------
    subvolume volumes/workspace
        group                   workspace
        snapshot_dir            snapshots/workspace
        target send-receive     /mnt/local/data/archive/laptop_one/workspace
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_one/workspace
            group                   mobiledata
            target_preserve         30d 24w 12m 8y

###################################################################################
volume ssh://server_one/mnt/btrfs-top-lvl/
    group                   all,cron,remote,ssh,server_one
    #send_compressed_data    no
    #send_protocol           1
    #-------------------------------------
    subvolume volumes/root
        group                   root
        snapshot_dir            snapshots/root
        target send-receive     /mnt/local/data/archive/server_one/root
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/server_one/root
            group                   mobiledata
            target_preserve         30d 8w 12m 3y
    #-------------------------------------
    subvolume volumes/home
        group                   home
        snapshot_dir            snapshots/home
        target send-receive     /mnt/local/data/archive/server_one/home
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/server_one/home
            group                   mobiledata
            target_preserve         30d 24w 12m 8y
    #-------------------------------------
    subvolume volumes/data
        group                   data
        snapshot_dir            snapshots/data
        target send-receive     /mnt/local/data/archive/server_one/data
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/server_one/data
            group                   mobiledata
            target_preserve         6d 8w 12m 2y

###################################################################################
volume ssh://laptop_two/mnt/btrfs-top-lvl/
    group                   all,cron,remote,ssh,laptop_two
    send_compressed_data    yes
    send_protocol           2
    #-------------------------------------
    subvolume @/.snapshots/1/snapshot
        group                   root
        snapshot_dir            @/.snapshots/root
        target send-receive     /mnt/local/data/archive/laptop_two/root
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_two/root
            group                   mobiledata
            target_preserve         30d 8w 12m 3y
    #-------------------------------------
    subvolume @/home
        group                   home
        snapshot_dir            @/.snapshots/home
        target send-receive     /mnt/local/data/archive/laptop_two/home
            group                   localdata
            target_preserve_min     latest
            target_preserve         2w
        target send-receive     /mnt/usb/mobiledata/snapshots/laptop_two/home
            group                   mobiledata
            target_preserve         30d 24w 12m 8y

###################################################################################
# Reverse Backups
###################################################################################
volume /mnt/usb/mobiledata
    group                   all,cron,reverse,mobiledata
    snapshot_create         no
    #-------------------------------------
    subvolume snapshots/desktop_two/root
        target send-receive     /mnt/local/data/archive/desktop_two/root
            target_preserve_min     latest
            target_preserve         2w
    #-------------------------------------
    subvolume snapshots/desktop_two/home
        target send-receive     /mnt/local/data/archive/desktop_two/home
            target_preserve_min     latest
            target_preserve         2w
    #-------------------------------------
    subvolume snapshots/desktop_two/data
        target send-receive     /mnt/local/data/archive/desktop_two/data
            target_preserve_min     latest
            target_preserve         2w
    #-------------------------------------
    subvolume snapshots/desktop_two/vm
        target send-receive     /mnt/local/data/archive/desktop_two/vm
            target_preserve_min     latest
            target_preserve         2w
    #-------------------------------------
    subvolume snapshots/desktop_two/workspace
        target send-receive     /mnt/local/data/archive/desktop_two/workspace
            target_preserve_min     latest
            target_preserve         2w
Massimo-B commented 1 year ago

In this setup, btrbk archive works like this: btrbk --progress -v archive /mnt/usb/mobiledata/snapshots/desktop_two/home/ /mnt/local/data/archive/desktop_two/home/ But that would require to manually call 'btrbk archive' for each subvolume line.