search

psy0rz / zfs_autobackup

ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable.
https://github.com/psy0rz/zfs_autobackup
GNU General Public License v3.0
583 stars 62 forks source link

Mismatch in incremental streams when updating backup #243

Closed zorbathut closed 7 months ago

zorbathut commented 7 months ago

I'm trying to update an existing backup and getting the following error:

cannot receive incremental stream: most recent snapshot of backup-b/tank does not match incremental source

The platform in question is a TrueNAS Scale system with a manual install of zfs_autobackup. I got this error a while back (I don't have a log from the last try, sorry) and decided to just update zfs_autobackup and redo the backup; I updated zfs_autobackup to v3.2.2 and did a new backup from scratch, but now that I'm trying to update the backup I'm getting the same error.

The only slightly odd thing I'm doing is mounting ZFS via Truecrypt (yes, I know I should be using ZFS built-in encryption, but this is a platform without AES hardware support and it's phenomenally slow).

This time, I did a backup, intentionally interrupted it midway through to test resuming (it worked fine), ended up interrupting it again midway through for a reboot (it resumed fine), finished it up, and ran it again to verify that I could update. I could! Then I did another update a few weeks later and ran into this problem again.

Debug log follows:

root@deus[~]# python -m zfs_autobackup.zfs_autobackup.ZfsAutobackup --encrypt -v backup-b backup-b --verbose --debug
  ZfsAutobackup.py v3.2 - (c)2022 E.H.Eefting (edwin@datux.nl)

  NOTE: Source and target are on the same host, excluding target-path from selection.

  Selecting dataset property : autobackup:backup-b
  Snapshot format            : backup-b-%Y%m%d%H%M%S
  Timezone                   : Local
  Hold name                  : zfs_autobackup:backup-b

  #### Source settings
  [Source] Keep the last 10 snapshots.
  [Source] Keep every 1 day, delete after 1 week.
  [Source] Keep every 1 week, delete after 1 month.
  [Source] Keep every 1 month, delete after 1 year.

  #### Selecting
# [Source] Getting selected datasets
# [Source] CMD    > (zfs get -t volume,filesystem -o name,value,source -H autobackup:backup-b)
# [Source] backup-b/tank: Excluded (path in exclude list)
# [Source] backup-b/tank/.system: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/cores: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/ctdb_shared_vol: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/glusterd: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/samba4: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/services: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Excluded (path in exclude list)
# [Source] backup-b/tank/.system/webui: Excluded (path in exclude list)
# [Source] backup-b/tank/borg: Excluded (path in exclude list)
# [Source] backup-b/tank/noressa: Excluded (path in exclude list)
# [Source] backup-b/tank/public: Excluded (path in exclude list)
# [Source] backup-b/tank/scratch: Excluded (path in exclude list)
# [Source] backup-b/tank/srv: Excluded (path in exclude list)
# [Source] backup-b/tank/zorba: Excluded (path in exclude list)
# [Source] tank: Checking if dataset is changed
  [Source] tank: Selected
# [Source] tank/.system: Checking if dataset is changed
  [Source] tank/.system: Selected
# [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Checking if dataset is changed
  [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Selected
# [Source] tank/.system/cores: Checking if dataset is changed
  [Source] tank/.system/cores: Selected
# [Source] tank/.system/ctdb_shared_vol: Checking if dataset is changed
  [Source] tank/.system/ctdb_shared_vol: Selected
# [Source] tank/.system/glusterd: Checking if dataset is changed
  [Source] tank/.system/glusterd: Selected
# [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Checking if dataset is changed
  [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Selected
# [Source] tank/.system/samba4: Checking if dataset is changed
  [Source] tank/.system/samba4: Selected
# [Source] tank/.system/services: Checking if dataset is changed
  [Source] tank/.system/services: Selected
# [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Checking if dataset is changed
  [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Selected
# [Source] tank/.system/webui: Checking if dataset is changed
  [Source] tank/.system/webui: Selected
# [Source] tank/borg: Checking if dataset is changed
  [Source] tank/borg: Selected
  [Source] tank/ix-applications: Excluded
  [Source] tank/ix-applications/catalogs: Excluded
  [Source] tank/ix-applications/default_volumes: Excluded
  [Source] tank/ix-applications/docker: Excluded
  [Source] tank/ix-applications/k3s: Excluded
  [Source] tank/ix-applications/k3s/kubelet: Excluded
  [Source] tank/ix-applications/releases: Excluded
# [Source] tank/noressa: Checking if dataset is changed
  [Source] tank/noressa: Selected
# [Source] tank/public: Checking if dataset is changed
  [Source] tank/public: Selected
  [Source] tank/reserve: Excluded
# [Source] tank/scratch: Checking if dataset is changed
  [Source] tank/scratch: Selected
# [Source] tank/srv: Checking if dataset is changed
  [Source] tank/srv: Selected
# [Source] tank/zorba: Checking if dataset is changed
  [Source] tank/zorba: Selected

  #### Snapshotting
# [Source] tank: Checking if filesystem exists: was forced to True
# [Source] tank: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank)
# [Source] tank: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank@backup-b-20240205131028 tank)
  [Source] tank: No changes since backup-b-20240205131028
# [Source] tank/.system: Checking if filesystem exists: was forced to True
# [Source] tank/.system: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system)
# [Source] tank/.system: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system@backup-b-20240205131028 tank/.system)
  [Source] tank/.system: No changes since backup-b-20240205131028
# [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Checking if filesystem exists: was forced to True
# [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5)
# [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5@backup-b-20240205131028 tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5)
  [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: No changes since backup-b-20240205131028
# [Source] tank/.system/cores: Checking if filesystem exists: was forced to True
# [Source] tank/.system/cores: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/cores)
# [Source] tank/.system/cores: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/cores@backup-b-20240113092606 tank/.system/cores)
  [Source] tank/.system/cores: No changes since backup-b-20240113092606
# [Source] tank/.system/ctdb_shared_vol: Checking if filesystem exists: was forced to True
# [Source] tank/.system/ctdb_shared_vol: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/ctdb_shared_vol)
# [Source] tank/.system/ctdb_shared_vol: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/ctdb_shared_vol@backup-b-20240113092606 tank/.system/ctdb_shared_vol)
  [Source] tank/.system/ctdb_shared_vol: No changes since backup-b-20240113092606
# [Source] tank/.system/glusterd: Checking if filesystem exists: was forced to True
# [Source] tank/.system/glusterd: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/glusterd)
# [Source] tank/.system/glusterd: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/glusterd@backup-b-20240113092606 tank/.system/glusterd)
  [Source] tank/.system/glusterd: No changes since backup-b-20240113092606
# [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Checking if filesystem exists: was forced to True
# [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5)
# [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5@backup-b-20240205131218 tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5)
# [Source] tank/.system/samba4: Checking if filesystem exists: was forced to True
# [Source] tank/.system/samba4: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/samba4)
# [Source] tank/.system/samba4: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/samba4@backup-b-20240205131028 tank/.system/samba4)
  [Source] tank/.system/samba4: No changes since backup-b-20240205131028
# [Source] tank/.system/services: Checking if filesystem exists: was forced to True
# [Source] tank/.system/services: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/services)
# [Source] tank/.system/services: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/services@backup-b-20240113092606 tank/.system/services)
  [Source] tank/.system/services: No changes since backup-b-20240113092606
# [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Checking if filesystem exists: was forced to True
# [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5)
# [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5@backup-b-20240205131218 tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5)
# [Source] tank/.system/webui: Checking if filesystem exists: was forced to True
# [Source] tank/.system/webui: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/.system/webui)
# [Source] tank/.system/webui: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/.system/webui@backup-b-20240113092606 tank/.system/webui)
  [Source] tank/.system/webui: No changes since backup-b-20240113092606
# [Source] tank/borg: Checking if filesystem exists: was forced to True
# [Source] tank/borg: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/borg)
# [Source] tank/borg: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/borg@backup-b-20240205131218 tank/borg)
# [Source] tank/noressa: Checking if filesystem exists: was forced to True
# [Source] tank/noressa: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/noressa)
# [Source] tank/noressa: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/noressa@backup-b-20240113092606 tank/noressa)
  [Source] tank/noressa: No changes since backup-b-20240113092606
# [Source] tank/public: Checking if filesystem exists: was forced to True
# [Source] tank/public: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/public)
# [Source] tank/public: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/public@backup-b-20240205131218 tank/public)
# [Source] tank/scratch: Checking if filesystem exists: was forced to True
# [Source] tank/scratch: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/scratch)
# [Source] tank/scratch: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/scratch@backup-b-20240205131218 tank/scratch)
  [Source] tank/scratch: No changes since backup-b-20240205131218
# [Source] tank/srv: Checking if filesystem exists: was forced to True
# [Source] tank/srv: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/srv)
# [Source] tank/srv: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/srv@backup-b-20240205131141 tank/srv)
  [Source] tank/srv: No changes since backup-b-20240205131141
# [Source] tank/zorba: Checking if filesystem exists: was forced to True
# [Source] tank/zorba: Getting snapshots
# [Source] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/zorba)
# [Source] tank/zorba: Getting bytes written since our last snapshot
# [Source] CMD    > (zfs get -H -ovalue -p written@tank/zorba@backup-b-20240205131028 tank/zorba)
  [Source] tank/zorba: No changes since backup-b-20240205131028
  [Source] Creating snapshots backup-b-20240205131241 in pool tank
# [Source] CMD    > (zfs snapshot tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5@backup-b-20240205131241 tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5@backup-b-20240205131241 tank/borg@backup-b-20240205131241 tank/public@backup-b-20240205131241)

  #### Target settings
  [Target] Keep the last 10 snapshots.
  [Target] Keep every 1 day, delete after 1 week.
  [Target] Keep every 1 week, delete after 1 month.
  [Target] Keep every 1 month, delete after 1 year.
  [Target] Receive datasets under: backup-b

  #### Synchronising
# [Target] backup-b: Checking if filesystem exists
# [Target] CMD    > (zfs list backup-b)
# Checking target names:
# [Source] tank: -> backup-b/tank
# [Source] tank/.system: -> backup-b/tank/.system
# [Source] tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5: -> backup-b/tank/.system/configs-5ece5c906a8f4df886779fae5cade8a5
# [Source] tank/.system/cores: -> backup-b/tank/.system/cores
# [Source] tank/.system/ctdb_shared_vol: -> backup-b/tank/.system/ctdb_shared_vol
# [Source] tank/.system/glusterd: -> backup-b/tank/.system/glusterd
# [Source] tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5: -> backup-b/tank/.system/rrd-5ece5c906a8f4df886779fae5cade8a5
# [Source] tank/.system/samba4: -> backup-b/tank/.system/samba4
# [Source] tank/.system/services: -> backup-b/tank/.system/services
# [Source] tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5: -> backup-b/tank/.system/syslog-5ece5c906a8f4df886779fae5cade8a5
# [Source] tank/.system/webui: -> backup-b/tank/.system/webui
# [Source] tank/borg: -> backup-b/tank/borg
# [Source] tank/noressa: -> backup-b/tank/noressa
# [Source] tank/public: -> backup-b/tank/public
# [Source] tank/scratch: -> backup-b/tank/scratch
# [Source] tank/srv: -> backup-b/tank/srv
# [Source] tank/zorba: -> backup-b/tank/zorba
# [Source] zpool tank: Getting zpool properties
# [Source] CMD    > (zpool get -H -p all tank)
# [Target] zpool backup-b: Getting zpool properties
# [Target] CMD    > (zpool get -H -p all backup-b)
  [Source] tank: sending to backup-b/tank
# [Target] backup-b/tank: Determining start snapshot
# [Target] backup-b/tank: Checking if filesystem exists
# [Target] CMD    > (zfs list backup-b/tank)
# [Target] backup-b/tank: Getting snapshots
# [Target] CMD    > (zfs list -d 1 -r -t snapshot -H -o name backup-b/tank)
# [Source] tank@backup-a-20240205131009: common snapshot
# [Target] backup-b/tank: Creating virtual target snapshots
# [Target] backup-b/tank: Getting zfs properties
# [Target] CMD    > (zfs get -H -o property,value -p all backup-b/tank)
# [Source] tank: Getting zfs properties
# [Source] CMD    > (zfs get -H -o property,value -p all tank)
# [Source] tank@backup-b-20240205131028: Transfer snapshot to backup-b/tank
  [Target] backup-b/tank@backup-b-20240205131028: receiving incremental
# [Source] CMD    > (zfs send -L zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Source] CMD    > (zfs send -e zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Source] CMD    > (zfs send -c zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Target] CMD    > (zfs recv -s zfs_autobackup_option_test)
# [Target] STDERR > cannot receive: failed to read from stream
# [Target] backup-b/tank@backup-b-20240205131028: Enabled resume support
# [Target] CMD    > (zfs send -L -v -P -p -i @backup-a-20240205131009 tank@backup-b-20240205131028) | (zfs recv -u -x keylocation -x pbkdf2iters -x keyformat -x encryption -v -s backup-b/tank)
# [Source] STDERR > incremental tank@backup-a-20240205131009    tank@backup-b-20240205131028    624
# [Source] STDERR > size        624
! [Target] STDERR > cannot receive incremental stream: most recent snapshot of backup-b/tank does not
! [Target] STDERR > match incremental source
! [Target] Command "zfs recv -u -x keylocation -x pbkdf2iters -x keyformat -x encryption -v -s backup-b/tank" returned exit code 1 (valid codes: [0])
! [Source] tank: FAILED: Last command returned error
  Debug mode, aborting on first error
! Exception: Last command returned error
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/zfs_autobackup/zfs_autobackup/ZfsAutobackup.py", line 547, in <module>
    cli()
  File "/root/zfs_autobackup/zfs_autobackup/ZfsAutobackup.py", line 542, in cli
    failed_datasets=ZfsAutobackup(sys.argv[1:], False).run()
  File "/root/zfs_autobackup/zfs_autobackup/ZfsAutobackup.py", line 497, in run
    fail_count = self.sync_datasets(
  File "/root/zfs_autobackup/zfs_autobackup/ZfsAutobackup.py", line 353, in sync_datasets
    source_dataset.sync_snapshots(target_dataset, show_progress=self.args.progress,
  File "/root/zfs_autobackup/zfs_autobackup/ZfsDataset.py", line 1106, in sync_snapshots
    source_snapshot.transfer_snapshot(target_snapshot, features=features,
  File "/root/zfs_autobackup/zfs_autobackup/ZfsDataset.py", line 734, in transfer_snapshot
    target_snapshot.recv_pipe(pipe, features=features, filter_properties=filter_properties,
  File "/root/zfs_autobackup/zfs_autobackup/ZfsDataset.py", line 677, in recv_pipe
    self.zfs_node.run(cmd, inp=pipe, valid_exitcodes=valid_exitcodes)
  File "/root/zfs_autobackup/zfs_autobackup/ExecuteNode.py", line 177, in run
    raise(ExecuteError("Last command returned error"))
zfs_autobackup.zfs_autobackup.ExecuteNode.ExecuteError: Last command returned error
root@deus[~]#

I don't personally see anything useful in here so I've attached a --debug-output log as well: zfsautobackup.txt

I am happy to troubleshoot as requested; let me know what would be useful to know and I'll go find it for you! But I really don't want to have to redo my backup from scratch every single time.

psy0rz commented 7 months ago

i suspect your common snapshot does not actually match.

check the guid of both common snapshots, or use the latest version in the master branch. (that will check the guid automaticly)

zorbathut commented 7 months ago

or use the latest version in the master branch. (that will check the guid automaticly)

This seems to have done it; it had about half an hour of copying to do and it was done, which is way less than starting from scratch (which would have been about three days).

I have no idea what the underlying issue was and if there's anything further you want me to do to troubleshoot, let me know, otherwise I'm calling this done. Thank you!

psy0rz commented 7 months ago

ah nice! did the new version warn you about a incompatible snapshot, which it then could delete for you?

zorbathut commented 7 months ago

Yep! Log snippet:

NOTE: [Target] backup-b/tank/borg@backup-a-20240205131009: Common snapshot has invalid guid, ignoring.
[Source] tank/borg@backup-b-20240117131045: Destroying
[Source] tank/borg@backup-b-20240123080223: Destroying
[Target] backup-b/tank/borg@backup-b-20240117131045: Destroying
[Target] backup-b/tank/borg@backup-b-20240123080223: Destroying
[Source] tank/borg@backup-b-20240205131028: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131141: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131204: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131218: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131241: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131714: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205131746: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205133033: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240205133323: -> backup-b/tank/borg
[Source] tank/borg@backup-b-20240206225456: -> backup-b/tank/borg
NOTE: [Target] backup-b/tank/public@backup-a-20240205131009: Common snapshot has invalid guid, ignoring.
[Source] tank/public@backup-b-20240122180205: Destroying
[Source] tank/public@backup-b-20240122180816: Destroying
[Source] tank/public@backup-b-20240122180938: Destroying
[Source] tank/public@backup-b-20240123014640: Destroying
[Source] tank/public@backup-b-20240123080223: Destroying
[Target] backup-b/tank/public@backup-b-20240122180205: Destroying
[Target] backup-b/tank/public@backup-b-20240122180816: Destroying
[Target] backup-b/tank/public@backup-b-20240122180938: Destroying
[Target] backup-b/tank/public@backup-b-20240123014640: Destroying
[Target] backup-b/tank/public@backup-b-20240123080223: Destroying
[Source] tank/public@backup-b-20240205131028: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131141: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131204: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131218: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131241: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131714: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205131746: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205133033: -> backup-b/tank/public
[Source] tank/public@backup-b-20240205133323: -> backup-b/tank/public
[Source] tank/public@backup-b-20240206225456: -> backup-b/tank/public
NOTE: [Target] backup-b/tank/scratch@backup-a-20240205131009: Common snapshot has invalid guid, ignoring.
[Source] tank/scratch@backup-b-20240122180205: Destroying
[Source] tank/scratch@backup-b-20240122180816: Destroying
[Source] tank/scratch@backup-b-20240122180938: Destroying
[Source] tank/scratch@backup-b-20240123014640: Destroying
[Source] tank/scratch@backup-b-20240123080223: Destroying
[Target] backup-b/tank/scratch@backup-b-20240122180205: Destroying
[Target] backup-b/tank/scratch@backup-b-20240122180816: Destroying
[Target] backup-b/tank/scratch@backup-b-20240122180938: Destroying
[Target] backup-b/tank/scratch@backup-b-20240123014640: Destroying
[Target] backup-b/tank/scratch@backup-b-20240123080223: Destroying
[Source] tank/scratch@backup-b-20240205131028: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205131141: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205131204: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205131218: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205131714: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205131746: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205133033: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240205133323: -> backup-b/tank/scratch
[Source] tank/scratch@backup-b-20240206225456: -> backup-b/tank/scratch
NOTE: [Target] backup-b/tank/srv@backup-a-20240205131009: Common snapshot has invalid guid, ignoring.
[Source] tank/srv@backup-b-20240205131028: -> backup-b/tank/srv
[Source] tank/srv@backup-b-20240205131141: -> backup-b/tank/srv
[Source] tank/srv@backup-b-20240205131714: -> backup-b/tank/srv
[Source] tank/srv@backup-b-20240205133033: -> backup-b/tank/srv
[Source] tank/srv@backup-b-20240206225456: -> backup-b/tank/srv

I'm reading this as "rewinding to something compatible, then going from there".

psy0rz commented 7 months ago

yes that is correct. :)