sebveit
commented
1 year ago It turnes out an incomplete zvol deletion was hindering the pool and zdb correctly asserts with (0x0 == 0x61).
One of my physical drives (ada3) was silently failing. After I've run several scrubs on the pool it finally signaled the failure via (ada3:ahcich3:0:0:0): CAM status: Command timeout in dmesg. smartctl -x /dev/ada3 showed 73 errors.
I've taken the failing device offline via zpool offline zdata ada3 and issued sysctl vfs.zfs.free_leak_on_eio=1 because of the permanent error regarding the destroyed zvol. After doing so, zdb executed normally and the affected pool showed no further checksum errors or events.
I might have lost some data (128M leaked) but I have to wait for a spare disk and then I'll figure it out.
As of now the affected pool looks like this:
# zpool status -v zdata
pool: zdata
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 08:48:29 with 0 errors on Fri Aug 4 17:49:56 2023
config:
NAME STATE READ WRITE CKSUM
zdata DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 OFFLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 ONLINE 0 0 0
ada6 ONLINE 0 0 0
ada7 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
<0xffffffffffffffff>:<0x1>zdb before (free_leak_on_eio=0):
# zdb -b -A zdata
Traversing all blocks to verify nothing leaked ...
loading concrete vdev 0, metaslab 115 of 116 ...
0 == bptree_iterate(spa->spa_meta_objset, spa->spa_dsl_pool->dp_bptree_obj, B_FALSE, count_block_cb, &zcb, NULL) (0x0 == 0x61)
6.94G completed (1169MB/s) estimated time remaining: 5hr 01min 49sec
17.8T completed (20581MB/s) estimated time remaining: 0hr 02min 00sec
leaked space: vdev 0, offset 0x14cfbe000, size 294912
…
leaked space: vdev 0, offset 0x1cfdfeb58000, size 11243520
block traversal size 19644664246272 != alloc 22224349704192 (leaked 2579685457920)
bp count: 113656434
ganged count: 0
bp logical: 14709804564480 avg: 129423
bp physical: 13932827621888 avg: 122587 compression: 1.06
bp allocated: 19644664246272 avg: 172842 compression: 0.75
bp deduped: 0 ref>1: 0 deduplication: 1.00
Normal class: 22224285573120 used: 70.31%
Embedded log class 64143360 used: 0.02%
additional, non-pointer bps of type 0: 30803
Dittoed blocks on same vdev: 116017async_destroy before (free_leak_on_eio=0):
# zpool get all zdata
...
zdata freeing 1.67T -
...
zdata leaked 0 -
...
zdata feature@async_destroy active localzdb after (free_leak_on_eio=1):
# zdb -b zdata
Traversing all blocks to verify nothing leaked ...
loading concrete vdev 0, metaslab 115 of 116 ...
17.8T completed (36129MB/s) estimated time remaining: 0hr 01min 06sec
leaked space: vdev 0, offset 0x40675cf0000, size 61440
leaked space: vdev 0, offset 0x1393e2f8e000, size 188743680
leaked space: vdev 0, offset 0x139407ac5000, size 61440
block traversal size 19644681965568 != alloc 19644870832128 (leaked 188866560)
bp count: 113656522
ganged count: 0
bp logical: 14709814276608 avg: 129423
bp physical: 13932836321792 avg: 122587 compression: 1.06
bp allocated: 19644681965568 avg: 172842 compression: 0.75
bp deduped: 0 ref>1: 0 deduplication: 1.00
Normal class: 22105317494784 used: 69.93%
Embedded log class 64561152 used: 0.02%
additional, non-pointer bps of type 0: 30809
Dittoed blocks on same vdev: 116054async_destroy after (free_leak_on_eio=1):
# zpool get all zdata
...
zdata freeing 0 -
...
zdata leaked 128M -
...
zdata feature@async_destroy enabled localI was getting massive amounts of checkum errors on zpool status -v zdata.
zpool events spamed every 5 seconds the same event but with different ena, time and eid:
# zpool events -v
Aug 4 2023 22:18:24.128208128 ereport.fs.zfs.data
class = "ereport.fs.zfs.data"
ena = 0x8754996b2f300801
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0x1095f287e8ca55f5
(end detector)
pool = "zdata"
pool_guid = 0x1095f287e8ca55f5
pool_state = 0x0
pool_context = 0x0
pool_failmode = "wait"
zio_err = 0x61
zio_flags = 0x808881
zio_stage = 0x1000000
zio_pipeline = 0x1080000
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x2
zio_objset = 0xffffffffffffffff
zio_object = 0x1
zio_level = 0x1
zio_blkid = 0x4a83
time = 0x64cd5d10 0x7a44d00
eid = 0x5566I hope that I'll get zfs to get rid of the permanent errors recorded by zpool status -v zdata, after I've resilvered the device. Maybe this information is helping someone else having similar issues.
This issue might be related to the following issues: https://github.com/openzfs/zfs/issues/3094#issuecomment-74444961, https://github.com/openzfs/zfs/issues/3108, https://github.com/openzfs/zfs/issues/8989#issuecomment-509229197, https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=multiple&id=218954
TLDR: Even if I was able to solve the problem, the pool leaked some space. In my case it was only 128 mb, so I can live with that. I'm still thinking that this is some kind of a bug, because ZFS went into a cycle that it couldn't break itself.
System information
Both pools (zroot and zdata) are not encrypted (no zfs builtin and/or geli). The system has ECC RAM.
Describe the problem you're observing
I observed a problem with one of my zvols (Permanent errors have been detected) right after a scrub. Since I don't need that zvol anymore I've decided to destroy it and started scrub after that, but this time I'm getting the same error with hex values.
After the scrub finished I rebooted the system and now I'm seeing a lot of checksum errors when utilizing
zpool status -vx(and no program or VM is accessing the pool).The scrub seemed to have repaired
2.73M in 09:42:34 with 0 errorsbut I'm still getting the mentioned error hint:Permanent errors have been detected in the following filesTrying to get more information via zdb fails when issuing the following command:
zdb -b zdataThe output of
dmesgseems to be fine with no errors right after system reboot. The only error message displayed is related tozdb:pid 93858 (zdb), jid 0, uid 0: exited on signal 6 (core dumped)Include any warning/errors/backtraces from the system logs