Possible way to identify partially failed, incomplete btrfs send's

[ThePythonicCow]

While developing some custom btrfs send/receive based backups of my own, I stumbled on to a way to create partially failed, incomplete, btrfs send's, and a way to identify the incomplete snapshots that result.

While doing my usual Google searching for discussion of such, I most often came upon this btrbk tool and discussions. So perhaps what I've noticed will be useful here as well, though more likely it's already well known to btrbk experts and I just haven't happened to come across it here.

To create partially failed send's, use the following shell script mem_lim_1G:

#!/bin/zsh
# Use cgroup memory control group to limit command to 1G resident memory

cgcreate -g memory:mem_lim_1G
cgset -r memory.limit_in_bytes=1G mem_lim_1G
exec cgexec -g memory:mem_lim_1G "$@"

Then prepend this mem_lim_1G command to the btrfs or btrbk invocation, as one would with the nice(1) or ionice(1) commands. Fine tune the 1G memory limit to be sufficient to allow the send to start, but insufficient to allow it to complete successfully. You'll notice one of your syslog files gets an entry such as "kernel: Memory cgroup out of memory: Kill process 9999 (yourbtrkbkcmd) score 0 or sacrifice child", and the btrfs send will die at that point.

To identify incomplete snapshots, such as result from the above hack, notice that the received_uuid field in the newly created snapshot on the receiving side will be set to "-", rather than to the proper uuid that it would have been set to, had the operation completed successfully.

The above was done on a 4.11.7-1-MANJARO Linux kernel, if that matters.

[ghost]

Is there a way to scan/check for corruptions in a received subvol? Would a scrub detect? Otherwise we have another important issue of truly "trusting" the backups since we have no good way of knowing the integrity of received send-receive. The logs, if any, needs to be very clear. Perhaps even a specific error_log is needed to highligt aborted or broken send-receive sessions.

[ThePythonicCow]

Is there a way to scan/check for corruptions in a received subvol?

Not an efficient way that I know of.

I've done a couple of brute force comparisons of the list of files, with their md5sum, from both the sending and receiving side, and not found any differences. But that sort of abandons any efficiencies that btrfs send/receive might offer.

Certainly the absence, so far as I am aware (which isn't far), of a clear description of the algorithm used by send/receive, of the data structures that algorithm uses, and of the extent, and limitations, of the robustness of this mechanism, is a concern in my view.

[digint]

Is there a way to scan/check for corruptions in a received subvol?

1. Use btrfs scrub periodically, it will find metadata checksum errors from faulty hardware. Running scrub periodically is also recommended by the btrfs devs.
2. You can run btrfs check to check the integrity more deeply. Not very practical as the filesystem must be unmounted
3. If you are paranoid (you should sometimes), check your backups with non-btrfs tools. I use rsync for this: rsync -a --delete -c -n -v --stats /path-to-snapshot/ /path-to-backup-subvolume/ (Note that if you have nested subvolumes (which are not copied), rsync will print them as directories missing on target). This is very slow as it reads all files and computes checksums on both sides, but that's exactly what we want to achieve here. Too bad it's interactive, you'll have to write a little script which parses the output to be able to run it in cron. If you know of another tool which does the same reliably, let me know!

Certainly the absence, so far as I am aware (which isn't far), of a clear description of the algorithm used by send/receive, of the data structures that algorithm uses, and of the extent, and limitations, of the robustness of this mechanism, is a concern in my view.

True, there's no much description (I'm aware of, some very low-level design here), but you can read the btrfs-progs code: cmds-receive.c. The idea is fairly simple, btrfs send creates a stream of system calls to be executed on the receive side, which results in "identical" data.

[ThePythonicCow]

Too bad it's interactive, you'll have to write a little script which parses the output to be able to run it in cron.

That's the sort of thing that I interactively do at a shell prompt.

For example, on the source system, run the commands cd /path-to-snapshot/ find . -type f -print | xargs md5sum | sort > /tmp/source.md5sums

Then on the target system (which might be the same system), run the commands cd /path-to-backup-subvolume/ find . -type f -print | xargs md5sum | sort > /tmp/target.md5sums

Then copy the /tmp/source.md5sums file to the target system (if they are different systems).

Then diff the two: diff /tmp/source.md5sums /tmp/target.md5sums

There should be no differences if the send/receive worked as intended.

This is all easy enough to put in a shell script, or to make a cron job of that shell script.

[ghost]

The md5sum script is useful. Perhaps it can be made as a feature in btrbk ? --verify-md5, --verify-sha256

Btrbk already knows what files were changed (btrbk diff) and could run checksum verification only on those.

[testbird]

https://github.com/csirac2/snazzer/#snazzer contains a command to "measure snapshots", i.e. calculate checksums, and found another reproducable inconsistency bug long ago.