mhx
commented
3 years ago That's awesome, thanks!
Haven't looked at the coredump yet, but I've got a few questions...
nabla@satella /media/veracrypt3/dwarfs $ doas mksquashfs /media/veracrypt3/dwarfs/mount/ /media/veracrypt2/LiterallyEverything-08-Jun-2021.sqsh -comp zstd -Xcompression-level 22 -b 1M
Parallel mksquashfs: Using 12 processors
Creating 4.0 filesystem on /media/veracrypt2/LiterallyEverything-08-Jun-2021.sqsh, block size 1048576.
[| ] 33715/7846602 0%
Read failed because Input/output errorAlso of note - these I/O errors are coming from a mounted DwarFS filesystem.
Even more bizarrely, despite both SquashFS and DwarFS failing consistently when trying to read roughly the same files, when I re-mounted the 2b2tca.dwarfs filesystem in question, I was able to read all of its content without any I/O errors at all. Truly baffling.
- So, the errors (SIGBUS, I/O error) happen when reading files from a mounted DwarFS image?
- When re-mounting the DwarFS image, these read errors go away?
- In #45, you've probably seen something similar happen reading from a mounted SquashFS image?
If this true, I wonder where the DwarFS/SquashFS images you're mounting are located. Are you accessing the images over NFS or some other network filesystem by any chance? (From the command line I guess the answer is "no". I don't know how veracrypt would fit into the whole equation.)
At least, if you're able to read the files after re-mounting, this would mean there's no issue with the consistency of the file system image.
Actually, I've looked at the coredump now, and it's consistent with your theory. The crash happens when trying to read the file (more specifically, when trying to compute the checksum — so it's also consistent with the crash from #45).
One more question: is the source DwarFS image using compressed metadata?
So here's my theory (which may very well also be flawed):
- All it takes to open a file on a mounted DwarFS image is metadata. As long as you never read from the file, no file data will ever be accessed in the DwarFS image.
- If the metadata in the image is compressed, it's decompressed to RAM upon mounting, so the metadata in the image file itself will never be accessed again.
- This likely means that when doing an
mmapon a file in the mounted DwarFS image, FUSE will not yet trigger a read. - Only upon accessing data in the
mmaped region, FUSE will trigger a read. This read fails, because for some reason thedwarfsprocess cannot access the DwarFS image anymore (though I'd expect to seedwarfsSIGBUS in this case) or it's reading junk from the image (no idea why that would happen, though). - The failing
readlikely causes FUSE to fail to populate themmaped region, which in turn causes a SIGBUS inmkdwarfs.
I'll definitely have to try and set something like this up. It's probably time to try out one of the fault-injecting FUSE drivers out there.
The mystery question that remains is: why do mounted DwarFS/SquashFS fail to read file data in the first place?
ghost
bionade24
The same issue, as described in issue #45, happened again. On the pre-compiled 0.5.5 release binaries. Twice, actually - the first time was on a completely different system, but the second time I was able to get a core dump. Even better, I actually think I know what's causing it.
When I ran
mksquashfsinstead ofmkdwarfson the exact same data, this happened:So, here's my theory (and I'm bad at these theories, so please take it with a grain of salt):
mksquashfsprobably just reads these files with regular oldopen()andread()calls, so whenever it encounters an I/O error, it can just skip the file and create an empty one as if nothing happened. Butmkdwarfs, as @mhx mentioned in the previous issue about this, makes extensive use ofmmap, so perhaps every time an I/O error occurs, the region of memory represented by the file it's trying to read becomes inaccessible, and aSIGBUSis triggered instead?Perhaps this SIGBUS could be caught, and behavior similar to
mksquashfscould be preserved whereby the file is simply skipped and replaced with an empty one, or maybe we could try re-reading the file several times before giving up and moving on instead?Also of note - these I/O errors are coming from a mounted DwarFS filesystem. When I got the SIGBUS error in #45, I was trying to read from a bunch of SquashFS filesystems, not a bunch of DwarFS filesystems, so this could be an issue with the physical disk (Although I still think DwarFS should definitely be robust enough to skip these errors rather than completely bailing out, as I uh... do actually need to recover these files)
Even more bizarrely, despite both SquashFS and DwarFS failing consistently when trying to read roughly the same files, when I re-mounted the
2b2tca.dwarfsfilesystem in question, I was able to read all of its content without any I/O errors at all. Truly baffling.Anyway, I have replied to the email I sent @mhx of the core dump for the previous issue with the new core dump (which is actually much smaller this time), so hopefully it's possible to figure out where this is specifically happening.