Allow transferring of single snapshots manually

[Massimo-B]

Hi, I want to transfer a snapshot to a different machine: btrbk archive -vv /mnt/snapshots/home/home.20220301T081224+0100/ ssh://mo2/mnt/archive/moblind But it doesn't do anything. I can't see any error.

On #btrbk@Libera we have discussed this some while ago. Your answer was:

Mo: btrbk archive operates on directories, not subvolumes

Why? I thought archive is used to transfer a backup (subvolume) to another location. The reason why I like to use btrbk archive instead of btrfs send|receive is, because btrbk may find some referenced parent on the target to use.

[Massimo-B]

Ok, one good explanation how to use archives with the current implementation is this: https://github.com/digint/btrbk/issues/90#issuecomment-223266380 However it would be fine to be able to use archive on whatever source subvolume to whatever btrfs location.

[digint]

I will need to refactor btrbk archive for this. I started working on this a while ago, but it's not finished yet...

Sorry for the late reply, was very busy lately...

[Massimo-B]

In advance, do you have some rough design how the new btrbk archive is going to be used and what will be new compared to the current implementation?

[digint]

Design goals for refactored btrbk archive are:

1. Use same code base as btrbk resume: currently archive follows its own code path, even if 90% is more or less identical to resume (especially with the improvements done for v0.32.0)
2. Allow configuring of archive sources. Many people want to be able to easily configure a snapshot directory for archiving, without a source subvolume to be defined. This is a bit cumbersome as subvolume section contains a snapshot_dir, which makes it hard to have a snapshot_dir without a subvolume...
3. Allow send-receive of non-btrbk snapshots. This will enable to copy filesystem trees recursively, as well as transferring single snapshote. Sophisticated functionality to find best parents/clone-src is in place, all I need to do here is find a decent way of combining archiving (which goes through the scheduler) and sending of non-btrbk subvolumes (which cannot go through the scheduler). This will probably be controlled through some include-non-btrbk-snapshots flag.

The first two points are almost finished (well, I will need to make sure that I did not drop any important features, and need to find a decent not-too-hacky way of enabling the config feature).

See progress on the archive-refactoring branch

[Massimo-B]

Now I'm looking for a way to clone a complete btrfs with btrbk snapshots. So instead of sending a single snapshot what this thread is about, I like to send all. Actually btrbk archive initially was meant to do that job, previously called btrbk clone: #79 But from what I learned, <target> cannot be something that is not configured, it cannot be some foreign target not mentioned in btrbk.conf. What is the minimal btrbk.conf for cloning one btrfs with lots of ro-snapshots to a new btrfs?

[digint]

But from what I learned, cannot be something that is not configured, it cannot be some foreign target not mentioned in btrbk.conf.

This is not true. The target directory for action "archive" can be any directory on a btrfs filesystem (there was a bug in btrbk-0.32.1 which caused subdir creation to fail, maybe you ran into that? Try btrbk-0.32.2, or if you are brave, the archive-refactoring branch)

Note that I'm making progress in refactoring: you might be interested in btrbk cp, which copies any read-only subvolume (recursively with -r,--recursive option. See the action-cp branch

[Massimo-B]

I try to understand, how btrbk archive is used today. In order to clone all snapshot I added a global

archive_preserve_min    all
archive_preserve        *d *w *m *y

to my config. Trying to attach my config here, but .txt is not allowed...I'll need my own repo here..

Anyway, this command seems to start a big transfer:

btrbk --progress -v archive /mnt/usb/mobiledata/snapshots/desk/ /mnt/usb/newmobiledata/snapshots/desk/

So this will just transfer all snapshots from this machine called "desk" from mobiledata to newmobile data. But I also have a lot of other /mnt/usb/mobiledata/snapshots/*, one for each machine beside this local machine. My current local machine's btrbk.conf only knows the /mnt/usb/mobiledata/snapshots/desk/ target, but how do I transfer all the other ones?

[Massimo-B]

Right as expected, doing the same command for a foreign machine doesn't work: btrbk --progress -v archive /mnt/usb/mobiledata/snapshots/lintop/ /mnt/usb/newmobiledata/snapshots/lintop/ because my local configuration doesn't know this target. The important information in the config actually are only these lines:

target send-receive     /mnt/usb/mobiledata/snapshots/desk/root
target send-receive     /mnt/usb/mobiledata/snapshots/desk/home
target send-receive     /mnt/usb/mobiledata/snapshots/desk/data
target send-receive     /mnt/usb/mobiledata/snapshots/desk/vm
...

However I was able to start the archive transfer for some other machine by creating a copy of the btrbk.conf and replacing all these lines with an other name, having a fake config like this:

target send-receive     /mnt/usb/mobiledata/snapshots/lintop/root
target send-receive     /mnt/usb/mobiledata/snapshots/lintop/home
...

Then the archive starts as usual. If simply faking the config makes the archive possible for other targets, then it can't be much effort to implement it in btrbk, no? For me it's not only about 2 archives, I need to create about 10 fake configs to get all the other snapshot lines transferred.

What is the current state of btrbk cp, can I already use it for my situation?

[luxagen]

Massimo, I really don't understand what you're seeing because I use btrbk archive literally all the time, without a single mention of the target drives in my btrbk.conf.

I'm convinced that something else is going on here. One thing I've noticed is that btrbk archive doesn't like relative source/target paths that start with . or .. for some reason (at least the last version I checked) and complains with a very cryptic error message.

As long as you use either absolute paths or relative paths that don't start that way, you shouldn't need to mention your targets in the configuration at all.