rincebrain
commented
7 years ago So, if we take as true that importing it read-only will reproducibly report a top-level checksum error without reporting an error on any particular disk, that would imply that the data is, in fact, "corrupt".
I'm slightly concerned that both your example pools are with the same type of disk, but let's put that aside for now.
What distro are you on, and what version of ZoL are you running now (cat /sys/module/zfs/version)?(Ideally, if you were installing it through a package manager and/or have syslog, what versions were you running in the past, and when did you update? The goal here would be to see if we can figure out what version you were running when you wrote the affected file on one of those pools, and see if there's any known explanation for munging data and generating a checksum error that's been fixed since.)
The problem with your theory about it only starting in July, presuming the data on-disk is actually "corrupt" and not an artifact of some recent code, is that your log on pool "o" says one or more blocks in the snapshot from June 28th are munged, which implies any such mangling code dates back at least that far.
...I'm slightly concerned that your corruption is happening only on what appear to be large disk image files. Are these pools primarily used for storing things like large disk images, or are most of the pools not occupied by disk images, and so this could possibly be an interesting data point?
dedup is on on one and off on another, so that's probably not it. mirror on one and raidz1 on the other, so it's probably not parity calculations. I suppose it could be some edge case for the optimized fletcher4 routines, but that could only matter if the mangling in the June 28th snapshot predated the dedup=sha256 setting, since there's no optimized sha256 committed that I can immediately see...plus, I'd expect bugs in that to have caused extreme screaming much sooner.
Just for completeness, though, what's /proc/cpuinfo say, and what's /sys/module/zcommon/parameters/zfs_fletcher_4_impl say?
JuliaVixen
kernelOfTruth
behlendorf
gordan-bobic
chrisrd
skrupler
jwittlincohen
tonyhutter
stephan2012
shodanshok
capnbb
ryao
I was backing up a pool (with
zfs send|recv), and it happened again. Two checksum errors on a raidz1 pool, with zero checksum errors on any device. Against my own paranoid judgement, I imported this pool read-write, rather than read-only like I usually do, so i can't reboot, swap the drives around, reimport it, and see if it faults on the same file at the same offset... but, I did do that last time, twice, and the fault was always in the exact same spot. I'm using ECC memory and I just tested it for several hours a few days ago. There are no error messages in Linuxdmesg, no SMART reports, everything claims to be functioning without error. (And these are not the crappy Seagate drives.)So... How did this checksum error happen?
About month ago, I copied most of the files off this pool with a regular
cp -avicopy, rather than azfs send|recv, there was no checksum error at that time. The disks have been mostly powered off since then, and I only plugged them back in today to make sure I really got all the data off them.So, what do I need to do with
zdbto debug this, and is anyone else noticing the same thing?I have about 50T of corrupt zpools now, which I need to "restore the entire pool from backup" because of checksums errors on the top level vdev, when the actual devices themselves are fine, and there's at least an extra disk of parity! How does this happen?
I think this has only started happening this month, so maybe there was a Github commit in July that did something?
Here's all the SMART stuff for these drives...
This is what the pool status looked like a few hours ago... It was created on the previous stable release version of ZFS, and I never upgraded it.
I have another pool which did this too. It had two checksum errors on the top level mirror, and zero checksum (or any kind of) errors on the individual drives. I ran a
zfs clearin the hope that it would forget about this checksum error until the next time the checksum really fails (to see if it was just a transient (cosmic ray) error or something). But it didn't forget about the I/O error on the file, but now it just says 0 errors... probably until I scrub it again... anyway... No hardware faults reported or detected....This checksum error also occurred when I was running the latest "master" version of ZFS and SPL from Github, as of about the end of July, beginning of Aug.