rincebrain
commented
2 years ago Yeah, see where it says
Jan 24 15:04:50 atuin kernel: [ 3740.917885] VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) failed (17179877376 <= 17179869184)
Jan 24 15:04:50 atuin kernel: [ 3740.919910] PANIC at space_map.c:405:space_map_load_callback()?
At that point, Progress Is Done - the kernel thread has died, it will not be coming back, and all locks that it was holding are forever taken.
Consequently, it's unsurprising that large swathes of things are not responsive once it happens.
(FWIW, I've never found -F/-X useful - I'm not sure how an automated emergency rewind has much worse success at this than manually doing it, but it is, and I've not looked into why. So you might want to experiment with grabbing a list of txgs still visible with e.g. zdb -lu and then trying manually specifying -T. You probably also want to adjust spa_load_verify_metadata if you don't want it to scan every single block for sanity checking before importing. Note that, as with -F and -X, this can be quite destructive, so, you know, last resort, make sure all the data you could get off is off or you have whole disk backups, etc.
It may still panic on a number of the possible -T values, so reboots between attempts may be necessary. I'd start at most recent and go backward. Normally I'd also suggest doing it with -o readonly=on, but it sounds like that works with the latest version too, so importing readonly in the past wouldn't be a strict improvement...could still try importing readonly before readwrite for each version though.)
e: Huh. #11923 looks like a similar case, on a much older Proxmox ZFS version, which got little attention. In #11691, they may have had hardware issues? Story unclear. #10942 was a reproducible bug with, IIUC, inconsistency between mirror legs being handled incorrectly, but seems to have been fixed, in theory, and the fix is in 2.1.0 and newer. (There are others, but that'll do as a starting point...)
techexo
stale[bot]
Kation
System information
Describe the problem you're observing
Hello all, I am running a proxmox server on a HP Proliant Microserver Gen8 with the latest kernel 5.13.19-3-pve. Since two days, I am unable to import my
discworldpool without crashing the server! It is a simple two-disks mirrored configuration.I was able to import the pool readonly (
zfs import -o readonly=on -d /dev/sdxx) and make a copy of all the data, I don't have however snapshots available. When trying to import the pool with write access, it hangs all zfs-related commands (zpool status, for example). I also triedzpool import -F -X -a -d /dev/disk/by-id, without much success either. The zpool import command is irresponsive and doesn't terminate to a SIGTERM or a SIGKILL. Looking at the disk activity with iostat doesn't show tremendous activity on the pool disks (but they are not dead as I have full read-only access).I'll join below the trace seen with dmesg. Any help understanding why I cannot import this pool will be much appreciated. Don't hesitate to ask me for more detailed information as I don't know what command outputs to send and help.
Thanks in advance
Describe how to reproduce the problem
Not sure it is reproducible on another environment than mine. On this server, trying to import the pool with write access is enough to reproduce the issue.
Include any warning/errors/backtraces from the system logs