backup test with extract / export-tar

ahjohannessen commented 2 years ago

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question

Your borg version (borg -V).

client: borg 1.2.1 server: borg1 1.1.17

Operating system (distribution) and version.

client: Ubuntu 20.04.4 LTS server: rsync.net / FreeBSD

Describe the problem you're observing.

I am having a bit of trouble doing continuous backup restore tests using extract / export-tar due to locks.

CRITICAL Failed to create/acquire the lock /data1/home/xyz/borg/bup01/hp03/lock (timeout).
...
Command '/usr/local/bin/borg prune --keep-minutely 0 --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 0 --keep-yearly 0 --prefix {hostname}- --remote-path /usr/local/bin/borg1/borg1 --info xyz@xyz.rsync.net:borg/bup01/hp03' returned non-zero exit status 2.

The problem I am observing is that I cannot do borg prune / borg compact / borg create / borg check (this is borgmatic default) while doing a borg extract / export-tar, because borg has a lock on the whole repo while performing a backup.

I can understand that prune/compact can conflict with export / export-tar, but a lock is also used during the other commands.

How am I supposed to test my backups if the test causes a backup to fail and a backup causes a backup test to fail? I have systemd timer that runs every 20 minutes.

Is this a case of using --bypass-lock while doing extraction?

ThomasWaldmann commented 2 years ago

borg always locks the repo when accessing it.

your problem is due to unsynchronised parallel access.

possible solutions:

use a script and just do borg create and the other borg ops you want to do sequentially
use e.g. --lock-wait 3600 to have borg wait up to an hour while trying to get a lock

ahjohannessen commented 2 years ago

@ThomasWaldmann As I use rsync.net, I get read-only .zfs snaphots folder with the borg repos in it. Perhaps I could do something with that for the backup restore test?

ThomasWaldmann commented 2 years ago

If it is read-only, borg won't be able to create a lock.

How about trying what I suggested above?

ahjohannessen commented 2 years ago

It is a nomad job on a different machine that does the extract and startup of a database, your suggestion assumes the same machine.

.zfs snapshot folder is read-only and immutable. Was thinking of using —bypass-lock with extract for the verification.

ThomasWaldmann commented 2 years ago

If you make sure the repo is read only and in a stable (snapshot) state, there is no risk trying --bypass-lock, but I would not recommend that if that is not the case.

ahjohannessen commented 2 years ago

I am sure, .zfs is a special folder with snapshots mounted as readonly. So I can do backup restore tests against the last daily snaphot and point borg to that for the repos and use the —bypass-lock flag with extract

ahjohannessen commented 2 years ago

@ThomasWaldmann Seems like doing something along these lines works:

snap="daily_$(date +%Y-%m-%d)"
repo="xyz@xyz.rsync.net:.zfs/snapshot/$snap/borg/abc/repo01"

# transient tmp as it is going to be a throw away docker container running this and repo path also changes each time.
export BORG_BASE_DIR=/tmp
borg --bypass-lock list --last 1 --short $repo | head -n1 | xargs -n1 -I% borg --bypass-lock extract $repo::%

I can do normal borgmatic runs every 20 min and do a full restore test daily using that .zfs snapshot folder trick. That gives me feedback that backups are working.

borgbackup / borg