This is probably an edge case, but either a fix or a warning in the documentation in e2fsck.conf might save someone some time. Failure first, then explanation:
e2fsck failed about 4 hours in, while also using /etc/e2fsck.conf containing only this:
[scratch_files]
directory = /big1
Here's the run:
# /usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck -vFftt /dev/mapper/fs
e2fsck 1.46.6-rc1 (12-Sep-2022)
Pass 1: Checking inodes, blocks, and sizes
Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x7f8f49ab3ffc
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck[0x436e80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f8e49687cb0]
/lib/x86_64-linux-gnu/libc.so.6(+0x907d5)[0x7f8e493477d5]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck[0x460ab2]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck[0x46214c]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck(ext2fs_tdb_store+0x548)[0x465678]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck[0x453106]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck(ext2fs_icount_store+0xb7)[0x453e67]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck(e2fsck_pass1+0x16c7)[0x41e7b7]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck(e2fsck_run+0x47)[0x415ca7]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck(main+0xed6)[0x411636]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f8e492d87ed]
/usr/local/bin/e2fsprogs-master/build/e2fsck/e2fsck[0x413b5d]
Command exited with non-zero status 8
I'd created that conf file because the machine this was running on had 16GB RAM and 2GB of swap and it OOM'ed shortly after starting Pass 2: Checking directory structure on the previous run. I strongly suspect that the failure above is because the inode file hit 2^32 bytes; I had a trivial bash loop printing out file sizes every 5 minutes and these are the iterations just before and just after the failure:
Earlier, both of those files started with TDB file\n and then had lots of binary. After the failure, the icount file did not contain that, but instead contained nothing but ASCII B except for the last four bytes: ^@^P^@^@. This also makes me believe this was a 2^32 wraparound followed by a scribble. (The scratch-file fs was a bog-standard, recently created ext4fs on a 64-bit machine and of course can thus store files bigger than 4GB and in fact contains only two other files at the moment, one of which is 3368GB.)
Removing the conf file and adding a 16GB swapfile succeeded, after using under 1 GB of the additional swapfile, so the very first OOM'ed run must have been very close:
As you've probably guessed, this is a filesystem containing a dirvish vault, hence all the hardlinks, and it's quite old and has been repeatedly expanded. (Last fsck was about six months ago just post-expansion; that expansion had required enabling 64bit. The fsck when I was done apparently fit in RAM because the fs was about 5% smaller then than it is now and I was apparently much closer to not fitting then I'd realized from the stats e2fsck prints.) This is also probably an edge case because a large fs like this would probably be running on a machine with either more than 16GB RAM or more swap and thus fsck might not hit the edge.
I can provide dumpe2fs -h output and OS info if necessary. (The OS is an old Ubuntu; hence the recently-built e2fsck instead of the distro default.)
One final issue: I noticed in perusing other open issues that you commented in https://github.com/tytso/e2fsprogs/issues/95 that there's a limit of 2^32 inodes. If this is independent of fs size, then I suspect I'm going to hit that wall if I ever try to double this fs again; this one currently has a bit more than 2^31 inodes. But I don't think I can change the bytes/inode ratio in a given fs, so doubling it would require making a new fs with a different ratio and copying this one to that---yet I know from experience that trying to copy such a heavily-crosslinked structure is extremely difficult due to memory exhaustion. Hmm.
This is probably an edge case, but either a fix or a warning in the documentation in e2fsck.conf might save someone some time. Failure first, then explanation:
e2fsck failed about 4 hours in, while also using
/etc/e2fsck.confcontaining only this:Here's the run:
I'd created that conf file because the machine this was running on had 16GB RAM and 2GB of swap and it OOM'ed shortly after starting
Pass 2: Checking directory structureon the previous run. I strongly suspect that the failure above is because the inode file hit 2^32 bytes; I had a trivial bash loop printing out file sizes every 5 minutes and these are the iterations just before and just after the failure:Earlier, both of those files started with
TDB file\nand then had lots of binary. After the failure, theicountfile did not contain that, but instead contained nothing but ASCIIBexcept for the last four bytes:^@^P^@^@. This also makes me believe this was a 2^32 wraparound followed by a scribble. (The scratch-file fs was a bog-standard, recently created ext4fs on a 64-bit machine and of course can thus store files bigger than 4GB and in fact contains only two other files at the moment, one of which is 3368GB.)Removing the conf file and adding a 16GB swapfile succeeded, after using under 1 GB of the additional swapfile, so the very first OOM'ed run must have been very close:
As you've probably guessed, this is a filesystem containing a dirvish vault, hence all the hardlinks, and it's quite old and has been repeatedly expanded. (Last fsck was about six months ago just post-expansion; that expansion had required enabling 64bit. The fsck when I was done apparently fit in RAM because the fs was about 5% smaller then than it is now and I was apparently much closer to not fitting then I'd realized from the stats
e2fsckprints.) This is also probably an edge case because a large fs like this would probably be running on a machine with either more than 16GB RAM or more swap and thus fsck might not hit the edge.I can provide
dumpe2fs -houtput and OS info if necessary. (The OS is an old Ubuntu; hence the recently-builte2fsckinstead of the distro default.)One final issue: I noticed in perusing other open issues that you commented in https://github.com/tytso/e2fsprogs/issues/95 that there's a limit of 2^32 inodes. If this is independent of fs size, then I suspect I'm going to hit that wall if I ever try to double this fs again; this one currently has a bit more than 2^31 inodes. But I don't think I can change the bytes/inode ratio in a given fs, so doubling it would require making a new fs with a different ratio and copying this one to that---yet I know from experience that trying to copy such a heavily-crosslinked structure is extremely difficult due to memory exhaustion. Hmm.