raw target: incremental backups without parents

calestyo commented 1 year ago

Hey.

This was with some older version (0.27.1 from Debian stable).

I first tested the raw dumps with full ones every 4 hours and incremental ones all the other hours, as described here https://github.com/digint/btrbk/issues/474#issuecomment-1312747334).

Until this works, I switched back now to non-incremental backups with this config:

transaction_log         /var/log/btrbk.log
snapshot_dir            snapshots/btrbk

snapshot_preserve_min       2d
snapshot_preserve       4d 2w 1m

target_preserve_min     latest
target_preserve         2d 1w 1m

ssh_identity            /etc/btrbk/ssh/id_ed25519
btrfs_commit_delete     each
timestamp_format        long-iso

raw_target_encrypt      gpg

gpg_keyring         /etc/btrbk/gnupg/pubring.kbx
gpg_recipient           backups@example.org

volume /data/system
    subvolume data
        target      raw ssh://backup1.example.org/var/local/lcg-backup/data/btrbk/source-host.example.org

So incremental was still yes per default, but in the systemd service I had overridden it to no.

Now first, I had a number of full dumps left, when I still did these all 4 hours:

-rw-r--r-- 1 root root 6,8G Nov 15 04:06 data.20221114T210053+0100.btrfs.gpg
-rw-r--r-- 1 root root  159 Nov 15 04:06 data.20221114T210053+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 6,8G Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg
-rw-r--r-- 1 root root  159 Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 6,5G Nov 15 04:31 data.20221115T040053+0100.btrfs.gpg
-rw-r--r-- 1 root root  159 Nov 15 04:31 data.20221115T040053+0100.btrfs.gpg.info

(no other dumps in the dir)

Then, with the above btrbk.conf I ran:

# btrbk resume

which of course lacked the override for incremental, with this output:

At subvol /data/system/snapshots/btrbk/data.20221113T000031+0100
At subvol /data/system/snapshots/btrbk/data.20221114T000050+0100
At subvol /data/system/snapshots/btrbk/data.20221115T063929+0100
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.27.1)

    Date:   Tue Nov 15 18:24:49 2022
    Config: /etc/btrbk/btrbk.conf
    Filter: target=backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org

Options:
    resume: No snapshots created

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/data/system/data
>>> backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org/data.20221113T000031+0100.btrfs.gpg
>>> backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org/data.20221114T000050+0100.btrfs.gpg
>>> backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org/data.20221115T063929+0100.btrfs.gpg
--- backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org/data.20221114T210053+0100.btrfs.gpg
--- backup1.example.org:/var/local/lcg-backup/data/btrbk/source-host.example.org/data.20221115T040053+0100.btrfs.gpg

Okay, so some of the full dumps got deleted, which is fine, as my retention policy is now 2d 1w 1m an before it was far smaller.

The end-result was this:

-rw-r--r-- 1 root root 178M Nov 15 18:24 data.20221113T000031+0100.btrfs.gpg
-rw-r--r-- 1 root root  217 Nov 15 18:24 data.20221113T000031+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 435M Nov 15 18:25 data.20221114T000050+0100.btrfs.gpg
-rw-r--r-- 1 root root  217 Nov 15 18:25 data.20221114T000050+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 6,8G Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg
-rw-r--r-- 1 root root  159 Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 100M Nov 15 18:25 data.20221115T063929+0100.btrfs.gpg
-rw-r--r-- 1 root root  217 Nov 15 18:25 data.20221115T063929+0100.btrfs.gpg.info

With these sidecars:

# Do not edit this file
TYPE=raw
FILE=data.20221113T000031+0100.btrfs.gpg
RECEIVED_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
RECEIVED_PARENT_UUID=58e32e3a-19e2-7644-8a84-d07526159f33
encrypt=gpg

-e -n #btrbk-v0.27.1
# Do not edit this file
TYPE=raw
FILE=data.20221114T000050+0100.btrfs.gpg
RECEIVED_UUID=912acaa5-9c1f-bf46-8779-268271c22323
RECEIVED_PARENT_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
encrypt=gpg

-e -n #btrbk-v0.27.1
# Do not edit this file
TYPE=raw
FILE=data.20221115T000053+0100.btrfs.gpg
RECEIVED_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
encrypt=gpg

-e -n #btrbk-v0.27.1
# Do not edit this file
TYPE=raw
FILE=data.20221115T063929+0100.btrfs.gpg
RECEIVED_UUID=a472b0bc-bffd-ed4e-8fa7-e3d832374335
RECEIVED_PARENT_UUID=2196e976-4594-214f-8f19-fffce618ee09
encrypt=gpg

As one can see, only the 3rd one (which is a full one, that exists still from the old config) is actually usable.

The first two ones were successfully transferred though they have no parent.
The last one, which I presume comes from my target_preserve_min latest, has no parent either.

Shouln't btrbk catch this and ... at least error out or just transfer a full send then or so? Or do I make some wrong assumptions?

Thanks, Chris.

calestyo commented 1 year ago

I should add, that other btrbk actions like clean don't seem to realise either, that these (incremental) backups are unusable.

calestyo commented 1 year ago

The same also happens with 0.32.5-1 but I've also noticed (at least with that version) the following:

While on the target host (the same from above), that already contained one full raw date, the same problem happens.
When I add another target host (with empty directory), then there, resume leads to a full dump first followed by incremental ones (all with a usable parent).

On backup2.example.org:

-rw-r--r-- 1 root root 6,5G Nov 15 19:03 data.20221113T000031+0100.btrfs.gpg
-rw-r--r-- 1 root root  220 Nov 15 19:03 data.20221113T000031+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 435M Nov 15 19:03 data.20221114T000050+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 19:03 data.20221114T000050+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 415M Nov 15 19:04 data.20221115T000053+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 19:04 data.20221115T000053+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 109M Nov 15 19:04 data.20221115T063929+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 19:04 data.20221115T063929+0100.btrfs.gpg.info

with:

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668534970
TYPE=raw
FILE=data.20221113T000031+0100.btrfs.gpg
RECEIVED_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668535382
INCOMPLETE=0

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668535382
TYPE=raw
FILE=data.20221114T000050+0100.btrfs.gpg
RECEIVED_UUID=912acaa5-9c1f-bf46-8779-268271c22323
RECEIVED_PARENT_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668535410
INCOMPLETE=0

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668535411
TYPE=raw
FILE=data.20221115T000053+0100.btrfs.gpg
RECEIVED_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
RECEIVED_PARENT_UUID=912acaa5-9c1f-bf46-8779-268271c22323
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668535444
INCOMPLETE=0

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668535444
TYPE=raw
FILE=data.20221115T063929+0100.btrfs.gpg
RECEIVED_UUID=a472b0bc-bffd-ed4e-8fa7-e3d832374335
RECEIVED_PARENT_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668535451
INCOMPLETE=0

I re-did backup1.example.org... first I removed all but the full one (with 6,8 GB) then resume:

-rw-r--r-- 1 root root 178M Nov 15 18:55 data.20221113T000031+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 18:55 data.20221113T000031+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 435M Nov 15 18:55 data.20221114T000050+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 18:55 data.20221114T000050+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 6,8G Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg
-rw-r--r-- 1 root root  159 Nov 15 04:18 data.20221115T000053+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 109M Nov 15 18:56 data.20221115T063929+0100.btrfs.gpg
-rw-r--r-- 1 root root  278 Nov 15 18:56 data.20221115T063929+0100.btrfs.gpg.info

with:

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668534936
TYPE=raw
FILE=data.20221113T000031+0100.btrfs.gpg
RECEIVED_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
RECEIVED_PARENT_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668534946
INCOMPLETE=0

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668534947
TYPE=raw
FILE=data.20221114T000050+0100.btrfs.gpg
RECEIVED_UUID=912acaa5-9c1f-bf46-8779-268271c22323
RECEIVED_PARENT_UUID=de6455ef-da07-c74a-ae9f-ff7c536b848b
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668534958
INCOMPLETE=0

-e -n #btrbk-v0.27.1
# Do not edit this file
TYPE=raw
FILE=data.20221115T000053+0100.btrfs.gpg
RECEIVED_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
encrypt=gpg

-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668534959
TYPE=raw
FILE=data.20221115T063929+0100.btrfs.gpg
RECEIVED_UUID=a472b0bc-bffd-ed4e-8fa7-e3d832374335
RECEIVED_PARENT_UUID=98b58b8e-793d-4f4e-ad44-c53cd9979fcb
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668534970
INCOMPLETE=0

That looks better now... though a bit mixed up... the first two now depend from the 3rd (the full one)... the 4th, also from the 3rd.

No idea, why that didn't work above, maybe there was actually something fixed after 0.27.1?

digint commented 1 year ago

No idea, why that didn't work above, maybe there was actually something fixed after 0.27.1?

In v0.32.0, incremental_prefs was implemented, which optimizes the selection of parents / clones, see btrbk.conf.5. I believe it should work fine with incremental raw targets, but this is not well tested (I don't use this feature any more, have migrated all backup targets to btrfs by now).

Sorry I don't have much time to dig deeper into this right now, as part of this code is also getting refactored in the action-cp branch which will make things even more transparent: the idea being that you could do something like btrbk cp --raw /source/* /raw-disk/.

I'm a bit puzzeled of your info files, this is wrong:

-e -n #btrbk-v0.32.5

The -e -n comes from the echo command. Do you have echo which does not support these options? Will have to look up what posix says about this, but I thought all coreutils (busybox included) supports it.

calestyo commented 1 year ago

In v0.32.0, incremental_prefs was implemented, which optimizes the selection of parents / clones, see btrbk.conf.5. I believe it should work fine with incremental raw targets

Ah,.. maybe that fixed it... (at least with this unusual "backward incremental send".
Do you think any of this could also fix my rotation issue with incremental dumps?

but this is not well tested (I don't use this feature any more, have migrated all backup targets to btrfs by now).

Well my main problem with that is security,... I'd like the backup host to be not able to read the backups.

Sorry I don't have much time to dig deeper into this right now, as part of this code is also getting refactored in the action-cp branch which will make things even more transparent: the idea being that you could do something like btrbk cp --raw /source/* /raw-disk/.

Okay... curious.

The -e -n comes from the echo command. Do you have echo which does not support these options? Will have to look up what posix says about this, but I thought all coreutils (busybox included) supports it.

POSIX says it's not portable, and actually "recommends" more or less against using echo but rather printf.

See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html

E.g. dash doesn't support it:

$ dash
$ echo -n -e foo
-e foo$

calestyo commented 1 year ago

Well my main problem with that is security,... I'd like the backup host to be not able to read the backups.

the raw dumps should in principle also allow running as a normal user without any sudo or so, which in reality still brings back root.

digint commented 1 year ago

@calestyo can we close this issue? I think the parent thing is correct in 0.32.5 (and no, I'm not going to debug old btrbk versions)

calestyo commented 1 year ago

It looked like it must have been fixed sometime after 0.27.1, so yes... I guess we can close.

Thanks.

digint / btrbk

raw target: incremental backups without parents #503