search

psy0rz / zfs_autobackup

ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable.
https://github.com/psy0rz/zfs_autobackup
GNU General Public License v3.0
583 stars 62 forks source link

Error while creating local snapshots: cannot receive resume stream: kernel modules must be upgraded to receive this stream #259

Open marceldegraaf opened 2 months ago

marceldegraaf commented 2 months ago

I've been using zfs_autobackup to create local snapshots for a while (from NVMe drives to a backup dataset on spinning disks) and I'm setting up remote synchronisation now, to make sure my ZFS snapshots are backed up remotely as well.

I have two datasets with the autobackup:local property set:

homes/db/dev-apps-pg15        autobackup:local  true              local
homes/home                    autobackup:local  true              local

And I use this command to create local snapshots to my tank/autobackup dataset. This used to work fine:

/root/.local/bin/zfs-autobackup -v --clear-mountpoint --keep-source="10,15min6h,1h1d,1d1w,1w1m,1m1y,1y5y" --keep-target="10,15min6h,1h1d,1d1w,1w1m,1m1y,1y5y" --snapshot-format "{}-%Y-%m-%d-%H:%M:%S" local tank/autobackup

However, since earlier today this error appears when running that command:

! [Target] STDERR > cannot receive resume stream: kernel modules must be upgraded to receive this stream.
! [Target] Command "zfs recv -u -o canmount=noauto -v -s tank/autobackup/homes/db/dev-apps-pg15" returned exit code 1 (valid codes: [0])
! [Source] homes/db/dev-apps-pg15: FAILED: Last command returned error

Does this happen because of some (unintended) change I made to the datasets or the data in them, or is this a bug? This is on the latest pre-release version:

# zfs-autobackup --version
zfs-autobackup v3.3-beta.3 - (c)2022 E.H.Eefting (edwin@datux.nl)

Here's the "SYNCHRONISING" portion of the output with --verbose --debug enabled:

  #### Synchronising
# [Target] tank/autobackup: Checking if dataset exists
# [Target] CMD    > (zfs list tank/autobackup)
# Checking target names:
# [Source] homes/db/dev-apps-pg15: -> tank/autobackup/homes/db/dev-apps-pg15
# [Source] homes/home: -> tank/autobackup/homes/home
# [Target] tank/autobackup/homes/db: Checking if dataset exists
# [Target] CMD    > (zfs list tank/autobackup/homes/db)
# [Source] zpool homes: Getting zpool properties
# [Source] CMD    > (zpool get -H -p all homes)
# [Target] zpool tank: Getting zpool properties
# [Target] CMD    > (zpool get -H -p all tank)
# [Source] homes/db/dev-apps-pg15: Getting zfs properties
# [Source] CMD    > (zfs get -H -o property,value -p all homes/db/dev-apps-pg15)
# [Target] tank/autobackup/homes/db/dev-apps-pg15: Determining start snapshot
# [Target] tank/autobackup/homes/db/dev-apps-pg15: Checking if dataset exists
# [Target] CMD    > (zfs list tank/autobackup/homes/db/dev-apps-pg15)
# [Target] tank/autobackup/homes/db/dev-apps-pg15: Getting snapshots
# [Target] CMD    > (zfs list -d 1 -r -t snapshot -H -o name tank/autobackup/homes/db/dev-apps-pg15)
# [Source] homes/db/dev-apps-pg15@local-2024-06-27-13:15:01: Getting zfs properties
# [Source] CMD    > (zfs get -H -o property,value -p all homes/db/dev-apps-pg15@local-2024-06-27-13:15:01)
# [Target] tank/autobackup/homes/db/dev-apps-pg15@local-2024-06-27-13:15:01: Getting zfs properties
# [Target] CMD    > (zfs get -H -o property,value -p all tank/autobackup/homes/db/dev-apps-pg15@local-2024-06-27-13:15:01)
# [Target] tank/autobackup/homes/db/dev-apps-pg15@local-2024-06-27-13:15:01: common snapshot
# [Target] tank/autobackup/homes/db/dev-apps-pg15@remote-20240627133001: Getting zfs properties
# [Target] CMD    > (zfs get -H -o property,value -p all tank/autobackup/homes/db/dev-apps-pg15@remote-20240627133001)
# [Target] tank/autobackup/homes/db/dev-apps-pg15: Creating virtual target snapshots
# [Target] tank/autobackup/homes/db/dev-apps-pg15: Getting zfs properties
# [Target] CMD    > (zfs get -H -o property,value -p all tank/autobackup/homes/db/dev-apps-pg15)
# [Source] CMD    > (zfs send -t 1-12f7f6ac0b-120-789c636064000310a501c49c50360710a715e5e7a69766a6304081b9d40759e6db5ffc15806c762475f94959a9c9250c0ccfb2218661c8a7a515a702e5191c18e0f26c48f2499525a9c52089379c58f597e4435ca1b64dc05ba2cddf3c00499e132c9f97989bcac090919f9b5aac9f92a49f925aa69b585050ac5b906e68ea90939f9c98a36b646064a26b60a66b64ae6b686c656c60656008f39704d41e5838a4e626a5a6e46783f9005e1b2a44 -n -v)
# [Source] homes/db/dev-apps-pg15@local-2024-06-27-13:30:01: resume token belongs to this snapshot
# [Source] homes/db/dev-apps-pg15@local-2024-06-27-13:30:01: Transfer snapshot to tank/autobackup/homes/db/dev-apps-pg15
  [Source] homes/db/dev-apps-pg15@local-2024-06-27-13:30:01: resuming
  [Source] homes/db/dev-apps-pg15@local-2024-06-27-13:30:01: -> tank/autobackup/homes/db/dev-apps-pg15
# [Source] CMD    > (zfs send -L zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Source] CMD    > (zfs send -e zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Source] CMD    > (zfs send -c zfs_autobackup_option_test)
# [Source] STDERR > cannot open 'zfs_autobackup_option_test': dataset does not exist
# [Target] CMD    > (zfs recv -s zfs_autobackup_option_test)
# [Target] STDERR > cannot receive: failed to read from stream
# [Target] tank/autobackup/homes/db/dev-apps-pg15@local-2024-06-27-13:30:01: Enabled resume support
# [Target] CMD    > (zfs send -L -e -v -P -t 1-12f7f6ac0b-120-789c636064000310a501c49c50360710a715e5e7a69766a6304081b9d40759e6db5ffc15806c762475f94959a9c9250c0ccfb2218661c8a7a515a702e5191c18e0f26c48f2499525a9c52089379c58f597e4435ca1b64dc05ba2cddf3c00499e132c9f97989bcac090919f9b5aac9f92a49f925aa69b585050ac5b906e68ea90939f9c98a36b646064a26b60a66b64ae6b686c656c60656008f39704d41e5838a4e626a5a6e46783f9005e1b2a44) | (zfs recv -u -o canmount=noauto -v -s tank/autobackup/homes/db/dev-apps-pg15)
# [Source] STDERR > resume token contents:
# [Source] STDERR > nvlist version: 0
# [Source] STDERR >     fromguid = 0x4ff4db031df01a37
# [Source] STDERR >     object = 0x16be6
# [Source] STDERR >     offset = 0x4000
# [Source] STDERR >     bytes = 0x9ec00
# [Source] STDERR >     toguid = 0x374f86184b10b626
# [Source] STDERR >     toname = homes/db/dev-apps-pg15@local-2024-06-27-13:30:01
# [Source] STDERR >     embedok = 1
# [Source] STDERR > incremental homes/db/dev-apps-pg15@local-2024-06-27-13:15:01    homes/db/dev-apps-pg15@local-2024-06-27-13:30:01    18446744073709314992
# [Source] STDERR > size    18446744073709314992
! [Target] STDERR > cannot receive resume stream: kernel modules must be upgraded to receive this stream.
! [Target] Command "zfs recv -u -o canmount=noauto -v -s tank/autobackup/homes/db/dev-apps-pg15" returned exit code 1 (valid codes: [0])
! [Source] homes/db/dev-apps-pg15: FAILED: Last command returned error
  Debug mode, aborting on first error
! Exception: Last command returned error
Traceback (most recent call last):
  File "/root/.local/bin/zfs-autobackup", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsAutobackup.py", line 574, in cli
    failed_datasets=ZfsAutobackup(sys.argv[1:], False).run()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsAutobackup.py", line 529, in run
    fail_count = self.sync_datasets(
                 ^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsAutobackup.py", line 390, in sync_datasets
    source_dataset.sync_snapshots(target_dataset, show_progress=self.args.progress,
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsDataset.py", line 1181, in sync_snapshots
    source_snapshot.transfer_snapshot(target_snapshot, features=features,
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsDataset.py", line 785, in transfer_snapshot
    target_snapshot.recv_pipe(pipe, features=features, filter_properties=filter_properties,
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ZfsDataset.py", line 699, in recv_pipe
    self.zfs_node.run(cmd, inp=pipe, valid_exitcodes=valid_exitcodes)
  File "/root/.local/pipx/venvs/zfs-autobackup/lib/python3.11/site-packages/zfs_autobackup/ExecuteNode.py", line 177, in run
    raise(ExecuteError("Last command returned error"))
zfs_autobackup.ExecuteNode.ExecuteError: Last command returned error
psy0rz commented 1 month ago

hmm thats strange, perhaps the resume status is incompatible due to an upgraded kernel?

Try aborting the interrupted backup with zfs recv -A tank/autobackup/homes/db/dev-apps-pg15