olidal
commented
3 years ago I have been investigating a bit this issue and came with a quick and maybe not so dirty fix.
First, looking at the code, I was not able to track the code path the produces the trace above. Ie. I didn't find how and why metaslab_alloc_dva did end up calling range_tree_remove. I am not familiar with zfs module debugging, so I probably missed something, eg. maybe a context switch?
Anyway, looking more closely at the error, it happens when an ASSERT detects an attempt to free a zero-sized segment. The question is then: how potentially harmful is this error, and does it need an ASSERT such taht the whole system is blocked.
My impression is that the caller (which I was not able to identify), skipped a zero-size test and didnt prevent the useless call. So I believe it is harmless to just return without blocking everything.
As a temporary mitigation, I tested the following fix, which replaces the original ASSERT with a call to zfs_panic_recover and activated the zfs_recover option, such that ZFS operations are not fully interrupted as with assert.
/*
* Turn this into a recoverable error.
* Requesting a null size allocation is certainly wrong, but there is a chance it
* only happens after a test was skipped somewhere that would have prevented to
* call this function.
*/
/* VERIFY3U(size, !=, 0); */
if ( size == 0 ) {
zfs_panic_recover("zfs: freeing 0-sized segment "
"(offset=%llu size=%llu)",
(longlong_t)start, (longlong_t)size);
return;
}As expected, when the situation occurs, I now have the following warning messages instead of the ASSERT induced PANIC
[14135.170214] WARNING: zfs: freeing 0-sized segment (offset=17179906048 size=0)
[14135.170231] WARNING: zfs: freeing 0-sized segment (offset=68719722496 size=0)I think this fix may be acceptable because, the severity of the error does not seem to deserve such a strong response as a PANIC. Still it allows for tracking the issue since it still produces a trace in logs.
This fix is still in operation (an will be as long as no better is found), do not hesitate to request action for investigating further.
SpComb
spmfox
nahall
devyn
Derkades
mafredri
Luflosi
thefloweringash
clefru
Majiir
System information
Describe the problem you're observing
After printing a panic message, all zfs operations are stuck, the machine needs to be rebooted. See below for panic message.
Note: This is NOT the zfs version that comes with the distribution. This is the locally recompiled zfs-0.8-release tagged version, but recompiled with NO error, using the default compilation settings.
Describe how to reproduce the problem
Hard to tell exactly when and why, but seems to happen under significant load, and most probably when trying to destroy snapshots. After a reboot, it happens again, not immediately, but unpredictably. The stack trace when it reproduces is identical.
I haven't tried yet locally compiled earlier versions, but it seems that earlier versions that came with proxmox kernel 5.4.78 (zfs version 0.8.5) didnt produce this error. (proxmox 6.1.1 or 6.2.1 fresh install without apt update I guess).
Note that it MAY have started after I had once to destroy 13k snapshots that had been accumulating in a dataset. But I don't see why it and I am not totally sure about the timing. Maybe completely unrelated.
Scrub after occurrence completed without error, and it happened again after scrubbing.
Note that most datasets are encrypted. This is a backup server, mostly receiving encrypted snapshots sent in raw replication differential streams. Source streams are also produced using a locally recompiled 0.8 version (I had to downgrade to this version because of another issue on ZFS version 2.0.X that I reported in a separate issue that totally brake our backup scripts).
Include any warning/errors/backtraces from the system logs