scratchings
commented
2 years ago Interesting observation. It panic'd again whilst I was preparing the above logs so I've just restarted again. Fresh system running z_upgrade only
zpool status -x
Shows data errors including
backup_pool/h9.local/fs2/fs8:<0x0>
By hand I run the recursive syncoid to backup_pool/h9.local/fs2....
sudo /usr/local/bin/syncoid --no-sync-snap --create-bookmark -r --skip-parent syncoid@h9.local:sfs1/sfs2 backup_pool/h9.local/fs2 NEWEST SNAPSHOT: autosnap_2022-01-18_12:30:01_hourly INFO: no snapshots on source newer than autosnap_2022-01-18_12:30:01_hourly on target. Nothing to do, not syncing. NEWEST SNAPSHOT: autosnap_2022-01-18_12:30:01_hourly INFO: no snapshots on source newer than autosnap_2022-01-18_12:30:01_hourly on target. Nothing to do, not syncing. NEWEST SNAPSHOT: autosnap_2022-01-18_12:30:01_hourly INFO: no snapshots on source newer than autosnap_2022-01-18_12:30:01_hourly on target. Nothing to do, not syncing. Resuming interrupted zfs send/receive from sfs1/sfs2/fs8 to backup_pool/h9.local/fs2/fs8 (~ 1.1 MB remaining): 495kiB 0:00:00 [4.18MiB/s] [=====================================================> ] 45%
Message from syslogd@myhost.local at Jan 18 13:22:21 ... kernel:VERIFY3(0 == dmu_bonus_hold_by_dnode(dn, FTAG, &db, flags)) failed (0 == 5)
Message from syslogd@fs3 at Jan 18 13:22:21 ... kernel:PANIC at dmu_recv.c:1811:receive_object()
Sorry, hadn't had chance to set the debug mode for the module.
Could these repeated crashes be due to hang overs from previous crashes and thus a multi-scrub needs to be run to completion to try and clear these? Where possible I've been deleting the 'corrupt' file systems, but in some cases we don't wish to loose the fs history. After a couple of scrubs (historically) we have seen all data 'errors' go away but as a scrub takes ca 3 days (so we'd be looking at a week of downtime) we haven't been doing this regularly.
rincebrain
arussell
lheckemann
zrav
clhedrick
xofyarg
PaulZ-98
ryao
siilike
System information
Describe the problem you're observing
When I start sending raw ZFS snapshots to a different system, my Linux systen (4.19.0-14-amd64) starts to hang completely. I can ping it, I can start a very commands (such as dmesg) but most commands hang (incl zfs, zpool, htop, ps, ...). The entire systems hangs completely.
Dmesg shows the following entries at the time of the occurance:
Interestingly, the transfer continues happily but just everything else in the system hangs. The only way to recover is resetting the machine (since not even reboot works).
Describe how to reproduce the problem
It's a tough one. It seems to me that the issue might be load related in some sense since it only occurs if I have two zfs send's (via syncoid) running in parallel that have to do with encrypted datasets.
Transfer 1
The first one sends datasets from an unecrypted dataset into an encrypted one (I migrate to encryption).
I use syncoid and use the command:
syncoid -r --skip-parent --no-sync-snap zpradix1imain/sys/vz zpradix1imain/sys/vz_encThis translates into
zfs send -I 'zpradix1imain/sys/vz/main'@'zfs-auto-snap_hourly-2021-03-02-1917' 'zpradix1imain/sys/vz/main'@'zfs-auto-snap_frequent-2021-03-02-1932' | mbuffer -q -s 128k -m 16M 2>/dev/null | pv -s 16392592 | zfs receive -s -F 'zpradix1imain/sys/vz_enc/main'Transfer 2
I transfer data from an encrypted dataset raw to a secondary server. The syncoid command is:
syncoid -r --skip-parent --no-sync-snap --sendoptions=w --exclude=zfs-auto-snap_hourly --exclude=zfs-auto-snap_frequent zpradix1imain/data root@192.168.200.12:zpzetta/radix/dataThis translates into:
zfs send -w 'zpradix1imain/data/home'@'vicari-prev' | pv -s 179222507064 | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null | ssh ...In summary: