dswartz
commented
9 years ago You got me concerned about my 3 pools on 2 servers, so I went to check them. Maybe I am out to lunch, but I can't get this to work at all:
[root@centos7-ha3 ~]# zpool status pool: tank-copy state: ONLINE scan: scrub repaired 0 in 1h44m with 0 errors on Sun Feb 15 01:44:07 2015 config:
NAME STATE READ WRITE CKSUM
tank-copy ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-5000c5002e38e0eb ONLINE 0 0 0
wwn-5000c5002e3aa680 ONLINE 0 0 0
logs
wwn-55cd2e404b4cd14f ONLINE 0 0 0errors: No known data errors
but:
[root@centos7-ha3 ~]# zdb -mc tank-copy zdb: can't open 'tank-copy': No such file or directory
this fails for all 3 pools. centos 7 using a code drop from back in early December. I checked and double-checked the command syntax from the man page and other folks emails. wth?
DeHackEd
sempervictus
behlendorf
dweeezil
gofman
kernelOfTruth
jay-to-the-dee
mooinglemur
kleini
mabod
bud4
Orfheo
dioni21
With all the changes and portage going on in the codebase, especially by people like me who insist on tanning their hide and sharpening teeth on the bleeding edge, there's a decent chance that spacemaps have suffered on more pools than people may realize. While digging through #3094 i'm seeing that i am not the only one who has spacemap refcount mismatch errors which are showing up in zdb -m, but not registering as errors in a scrub. As @behlendorf pointed out, this is far from good since
I highly encourage anyone reading this to check their pools with zdb -m
Since scrubs dont flag this, and a quick grep through the git logs shows a bunch of space map related changes since ZoL went 0.6.2, there's a decent chance that people other than the "adventurous" lot who used #2909 may be looking at similar issues.
To resolve this, i suggest we teach scrub, either implicitly, or through a CLI flag, to keep track of data and metadata sizing as it traverses the metaslabs (used space accounting). It should occasionally compare this with the space map, and upon deviation recompute the space map for every metaslab affected. The new space maps should only commit once the blocks they describe have been verified. Scrub should also probably ring some alarm bells when it detects this condition, as i imagine that a write to a block which the SM presented as available will actually look valid despite having overwritten existing data since there's a valid pointer to it in the tree.
With the on-disk format changing all the time, and the data structures becoming more complex and interdependent, it may be worth revisiting what scrub should be doing, as opposed to what it does today.