boomshroom
commented
2 months ago Checking bcachefs fs usage on the filesystem revealed that two of my devices are marked as having 16EiB of need_discard, with the exact number of buckets listed being exactly 1 apart. Because of this the gc got upset thinking the alloc keys being written should've had a data_type of needs_discard when elsewhere they were being set to free. This seems to be the closest thing to a root cause that I've been able to find so far.
Edit: Got fsck to do a btree node scan. This might work.
Edit2: overlapping btree nodes with same seq! halting That seems like a known limitation that's being worked on. I might just have to wait for an update. Worth mentioning is both found btree nodes have identical ranges starting from POS_MIN, rather than just overlapping. One also has 3 replicas while the other only has 1 (likely due to often running fsck with data_replicas and metadata_replicas at 1 to minimise the chance that it would fail to write to all disks, and running bcachefs data rereplicate once it was running again).
koverstreet
Fsck log
``` mounting version 1.6: btree_subvolume_children opts=ro,errors=continue,metadata_checksum=xxhash,data_checksum=xxhash,background_compression=zstd:15,metadata_target=/dev//dev/sda,foreground_target=/dev//dev/sda,background_target=hdd,promote_target=ssd,degraded,fsck,fix_errors=yes,ratelimit_errors,nochanges,read_only,reconstruct_alloc recovering from unclean shutdown starting journal read journal read done on device /dev/nvme0n1p3, ret 0 journal read done on device /dev/sda, ret 0 journal read done on device /dev/sdc, ret 0 journal read done on device /dev/sdb, ret 0 journal read done, replaying entries 144314248-144318354 Journal keys: 5019139 read, 4997452 after sorting and compacting dropping and reconstructing all alloc info alloc_read... done stripes_read... done snapshots_read... done check_allocations...pointer to missing indirect extent u64s 7 type reflink_p 808630793:8:4294967293 len 8 ver 0: idx 449368096 front_pad 0 back_pad 0 missing range 449368096-449368104, fixing pointer to missing indirect extent u64s 7 type reflink_p 1075323961:8:4294967293 len 8 ver 0: idx 449368104 front_pad 0 back_pad 0 missing range 449368104-449368112, fixing pointer to missing indirect extent u64s 7 type reflink_p 1211997419:16:4294967293 len 16 ver 0: idx 449368000 front_pad 0 back_pad 0 missing range 449368000-449368016, fixing bucket 1:1445838 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613554810:16:4294967287 len 16 ver 0: durability: 2 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress none ptr: 1:1445838:152 gen 0 cached crc: c_size 8 size 16 offset 0 nonce 0 csum xxhash compress zstd ptr: 2:3768407:392 gen 2 ptr: 3:4567429:296 gen 2, fixing bucket 1:1445837 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613554842:16:4294967287 len 16 ver 0: durability: 2 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress none ptr: 1:1445837:360 gen 0 cached crc: c_size 8 size 16 offset 0 nonce 0 csum xxhash compress zstd ptr: 3:4540806:256 gen 11 ptr: 2:3765849:424 gen 1, fixing bucket 0:1400357 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613649326:88:4294967280 len 88 ver 0: durability: 2 crc: c_size 88 size 88 offset 0 nonce 0 csum xxhash compress none ptr: 0:1400357:424 gen 0 cached crc: c_size 16 size 88 offset 0 nonce 0 csum xxhash compress zstd ptr: 2:3758431:216 gen 11 ptr: 3:4582633:488 gen 2, fixing bucket 1:1433418 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613657840:40:4294967293 len 40 ver 0: durability: 2 crc: c_size 40 size 40 offset 0 nonce 0 csum xxhash compress none ptr: 1:1433418:400 gen 0 cached crc: c_size 16 size 40 offset 0 nonce 0 csum xxhash compress zstd ptr: 3:4570426:56 gen 4 ptr: 2:3765836:440 gen 4, fixing bucket 1:1433418 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613657849:8:4294967293 len 8 ver 0: durability: 2 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress none ptr: 1:1433418:440 gen 0 cached ptr: 3:4567699:0 gen 1 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 2:3765836:456 gen 4, fixing bucket 1:1433457 different types of data in same bucket: btree, user while marking u64s 11 type extent 1613657861:8:4294967293 len 8 ver 0: durability: 2 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress none ptr: 3:4573932:288 gen 2 ptr: 2:3768589:360 gen 0 ptr: 1:1433457:416 gen 0 cached rebalance: target hdd compression zstd:15, fixing bucket 1:1433457 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613657862:24:4294967293 len 24 ver 0: durability: 2 crc: c_size 24 size 24 offset 0 nonce 0 csum xxhash compress none ptr: 1:1433457:424 gen 0 cached crc: c_size 8 size 24 offset 0 nonce 0 csum xxhash compress zstd ptr: 3:4573932:296 gen 2 ptr: 2:3768589:368 gen 0, fixing bucket 1:1433457 different types of data in same bucket: btree, user while marking u64s 13 type extent 1613657862:88:4294967293 len 64 ver 0: durability: 2 crc: c_size 64 size 64 offset 0 nonce 0 csum xxhash compress none ptr: 1:1433457:448 gen 0 cached ptr: 3:4567718:0 gen 1 rebalance: target hdd compression zstd:15 crc: c_size 32 size 64 offset 0 nonce 0 csum xxhash compress zstd ptr: 2:3768589:376 gen 0, fixing bucket 1:1445781 different types of data in same bucket: btree, user while marking u64s 11 type extent 1613676347:8:4294967287 len 8 ver 0: durability: 2 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress none ptr: 3:1133724:40 gen 6 ptr: 2:3768936:328 gen 2 ptr: 1:1445781:176 gen 0 cached rebalance: target hdd compression zstd:15, fixing bucket 0:1411344 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613676465:16:4294967287 len 16 ver 0: durability: 2 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress none ptr: 0:1411344:360 gen 1 cached crc: c_size 8 size 16 offset 0 nonce 0 csum xxhash compress zstd ptr: 3:4556077:296 gen 3 ptr: 2:3756872:488 gen 9, fixing bucket 0:1411353 different types of data in same bucket: btree, user while marking u64s 12 type extent 1613676466:1048:4294967287 len 16 ver 0: durability: 2 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress none ptr: 3:4569340:24 gen 2 ptr: 0:1411353:248 gen 1 cached crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress incompressible ptr: 2:3748347:112 gen 13, fixing Ratelimiting new instances of previous error pointer to missing indirect extent u64s 7 type reflink_p 1613688053:449000:4294967293 len 176 ver 0: idx 448998136 front_pad 0 back_pad 0 missing range 448998248-448998304, fixing pointer to missing indirect extent u64s 7 type reflink_p 1613688053:449000:4294967293 len 176 ver 0: idx 448998136 front_pad 0 back_pad 0 missing range 448998304-448998312, fixing pointer to missing indirect extent u64s 7 type reflink_p 1613688053:449000:4294967293 len 176 ver 0: idx 448998136 front_pad 0 back_pad 0 missing range 448998248-448998304, fixing pointer to missing indirect extent u64s 7 type reflink_p 1613688053:449000:4294967293 len 176 ver 0: idx 448998136 front_pad 0 back_pad 0 missing range 448998304-448998312, fixing pointer to missing indirect extent u64s 7 type reflink_p 1613692137:8:4294967293 len 8 ver 0: idx 449368056 front_pad 0 back_pad 0 missing range 449368056-449368064, fixing pointer to missing indirect extent u64s 7 type reflink_p 1747840859:8:4294967293 len 8 ver 0: idx 449368080 front_pad 0 back_pad 0 missing range 449368080-449368088, fixing pointer to missing indirect extent u64s 7 type reflink_p 1881970634:16:4294967293 len 16 ver 0: idx 449368016 front_pad 0 back_pad 0 missing range 449368016-449368032, fixing pointer to missing indirect extent u64s 7 type reflink_p 2016694364:8:4294967293 len 8 ver 0: idx 449368072 front_pad 0 back_pad 0 missing range 449368072-449368080, fixing Ratelimiting new instances of previous error Second GC pass needed, restarting: bucket 3:4484789 different types of data in same bucket: btree, user while marking u64s 9 type extent 1881478156:24:4294967293 len 24 ver 0: durability: 2 crc: c_size 16 size 24 offset 0 nonce 0 csum xxhash compress zstd ptr: 3:4484789:88 gen 2 ptr: 2:524295:32 gen 12, fixing bucket 3:4484789 different types of data in same bucket: btree, user while marking u64s 11 type reflink_v 0:174739872:0 len 16 ver 0: refcount: 1 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:4484789:40 gen 2 ptr: 2:516285:408 gen 1, fixing bucket 3:4484789 different types of data in same bucket: btree, user while marking u64s 11 type reflink_v 0:174739880:0 len 8 ver 0: refcount: 1 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:4484789:56 gen 2 ptr: 2:516285:424 gen 1, fixing bucket 3:4484789 different types of data in same bucket: btree, user while marking u64s 11 type reflink_v 0:174739896:0 len 16 ver 0: refcount: 1 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 16 size 16 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:4484789:64 gen 2 ptr: 2:516285:432 gen 1, fixing bucket 0:443029 gen 11 data type user has wrong cached_sectors: got 0, should be 40, fixing bucket 0:525490 gen 11 data type user has wrong cached_sectors: got 0, should be 8, fixing bucket 0:542305 gen 8 data type user has wrong cached_sectors: got 0, should be 80, fixing bucket 0:646507 gen 9 data type user has wrong cached_sectors: got 0, should be 88, fixing bucket 0:649784 gen 9 data type user has wrong cached_sectors: got 0, should be 8, fixing bucket 0:1226655 gen 15 data type user has wrong cached_sectors: got 0, should be 64, fixing bucket 0:1398708 gen 5 data type user has wrong cached_sectors: got 0, should be 16, fixing bucket 0:1410990 gen 7 data type user has wrong cached_sectors: got 0, should be 64, fixing bucket 1:586772 gen 5 data type user has wrong cached_sectors: got 0, should be 56, fixing bucket 1:604810 gen 6 data type user has wrong cached_sectors: got 0, should be 8, fixing bucket 1:652499 gen 10 data type user has wrong cached_sectors: got 0, should be 32, fixing Ratelimiting new instances of previous error reflink key has wrong refcount: u64s 11 type reflink_v 0:5779464:0 len 8 ver 0: refcount: 2 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:152 gen 3 ptr: 2:11194:416 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779544:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:232 gen 3 ptr: 2:11194:496 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779552:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:240 gen 3 ptr: 2:11194:504 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779560:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:248 gen 3 ptr: 2:11195:0 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779568:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:256 gen 3 ptr: 2:11195:8 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779576:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:264 gen 3 ptr: 2:11195:16 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779584:0 len 8 ver 0: refcount: 3 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:272 gen 3 ptr: 2:11195:24 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779672:0 len 8 ver 0: refcount: 2 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:360 gen 3 ptr: 2:11195:112 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779680:0 len 8 ver 0: refcount: 2 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:368 gen 3 ptr: 2:11195:120 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779704:0 len 8 ver 0: refcount: 2 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:3085975:392 gen 3 ptr: 2:11195:144 gen 1 should be 1, fixing reflink key has wrong refcount: u64s 11 type reflink_v 0:5779784:0 len 8 ver 0: refcount: 2 durability: 2 rebalance: target hdd compression zstd:15 crc: c_size 8 size 8 offset 0 nonce 0 csum xxhash compress incompressible ptr: 3:2592972:0 gen 0 ptr: 2:11195:224 gen 1 should be 1, fixing Ratelimiting new instances of previous error done going read-write journal_replay...invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:146045272:0 len 48 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:146045272:0 len 48 ver 0 new u64s 5 type error 0:146045272:0 len 48 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:146064992:0 len 128 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:146064992:0 len 128 ver 0 new u64s 5 type error 0:146064992:0 len 128 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:173640424:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:173640424:0 len 16 ver 0 new u64s 5 type error 0:173640424:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:177164152:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:177164152:0 len 16 ver 0 new u64s 5 type error 0:177164152:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:178046272:0 len 24 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:178046272:0 len 24 ver 0 new u64s 5 type error 0:178046272:0 len 24 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:203107112:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:203107112:0 len 16 ver 0 new u64s 5 type error 0:203107112:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:205612560:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:205612560:0 len 16 ver 0 new u64s 5 type error 0:205612560:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:230718048:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:230718048:0 len 8 ver 0 new u64s 5 type error 0:230718048:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:437915328:0 len 56 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:437915328:0 len 56 ver 0 new u64s 5 type error 0:437915328:0 len 56 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:441225328:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:441225328:0 len 8 ver 0 new u64s 5 type error 0:441225328:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:446604144:0 len 96 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:446604144:0 len 96 ver 0 new u64s 5 type error 0:446604144:0 len 96 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:446646560:0 len 24 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:446646560:0 len 24 ver 0 new u64s 5 type error 0:446646560:0 len 24 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199152:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199152:0 len 8 ver 0 new u64s 5 type error 0:447199152:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199160:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199160:0 len 8 ver 0 new u64s 5 type error 0:447199160:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199168:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199168:0 len 8 ver 0 new u64s 5 type error 0:447199168:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199176:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199176:0 len 8 ver 0 new u64s 5 type error 0:447199176:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199184:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199184:0 len 8 ver 0 new u64s 5 type error 0:447199184:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199192:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199192:0 len 8 ver 0 new u64s 5 type error 0:447199192:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447199264:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447199264:0 len 16 ver 0 new u64s 5 type error 0:447199264:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447236112:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447236112:0 len 8 ver 0 new u64s 5 type error 0:447236112:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326008:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326008:0 len 8 ver 0 new u64s 5 type error 0:447326008:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326016:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326016:0 len 8 ver 0 new u64s 5 type error 0:447326016:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326352:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326352:0 len 16 ver 0 new u64s 5 type error 0:447326352:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326496:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326496:0 len 8 ver 0 new u64s 5 type error 0:447326496:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326592:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326592:0 len 8 ver 0 new u64s 5 type error 0:447326592:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326648:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326648:0 len 8 ver 0 new u64s 5 type error 0:447326648:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326656:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326656:0 len 8 ver 0 new u64s 5 type error 0:447326656:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326832:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326832:0 len 8 ver 0 new u64s 5 type error 0:447326832:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447326904:0 len 24 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447326904:0 len 24 ver 0 new u64s 5 type error 0:447326904:0 len 24 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447327456:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447327456:0 len 16 ver 0 new u64s 5 type error 0:447327456:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447327472:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447327472:0 len 16 ver 0 new u64s 5 type error 0:447327472:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447328160:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447328160:0 len 8 ver 0 new u64s 5 type error 0:447328160:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447328336:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447328336:0 len 8 ver 0 new u64s 5 type error 0:447328336:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447328344:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447328344:0 len 8 ver 0 new u64s 5 type error 0:447328344:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447330680:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447330680:0 len 8 ver 0 new u64s 5 type error 0:447330680:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447331472:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447331472:0 len 8 ver 0 new u64s 5 type error 0:447331472:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447331480:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447331480:0 len 8 ver 0 new u64s 5 type error 0:447331480:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447331496:0 len 16 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447331496:0 len 16 ver 0 new u64s 5 type error 0:447331496:0 len 16 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447333208:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447333208:0 len 8 ver 0 new u64s 5 type error 0:447333208:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447333216:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447333216:0 len 8 ver 0 new u64s 5 type error 0:447333216:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447333224:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447333224:0 len 8 ver 0 new u64s 5 type error 0:447333224:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447340752:0 len 144 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447340752:0 len 144 ver 0 new u64s 5 type error 0:447340752:0 len 144 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447340776:0 len 24 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447340776:0 len 24 ver 0 new u64s 5 type error 0:447340776:0 len 24 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:447343016:0 len 40 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:447343016:0 len 40 ver 0 new u64s 5 type error 0:447343016:0 len 40 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:449343552:0 len 8 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:449343552:0 len 8 ver 0 new u64s 5 type error 0:449343552:0 len 8 ver 0 invalid bkey on insert from bch2_journal_replay -> 0x556d967a6e69s u64s 5 type error 0:146045272:0 len 48 ver 0 invalid key type for btree reflink (error) transaction updates for bch2_journal_replay journal seq 4294967295 update: btree=reflink cached=0 0x556d967a6e69S old u64s 5 type error 0:146045272:0 len 48 ver 0 new u64s 5 type error 0:146045272:0 len 48 ver 0 bch2_journal_replay(): error while replaying key at btree reflink level 0: Invalid argument bch2_journal_replay(): error Invalid argument bch2_fs_recovery(): error Invalid argument bch2_fs_start(): error starting filesystem Invalid argument shutting down going read-only finished waiting for writes to stop flushing journal and stopping allocators, journal seq 144329934 flushing journal and stopping allocators complete, journal seq 144329934 done going read-only, filesystem not clean shutdown complete ````bcachefs show-super /dev/sda`
``` Device: WDC WDS500G2B0A External UUID: 4d4903de-99c8-41a1-ab6a-83431f05b0dc Internal UUID: e6a58fab-3bb3-477a-8f1f-256d9d85f946 Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef Device index: 1 Label: Version: 1.6: btree_subvolume_children Version upgrade complete: 1.4: member_seq Oldest version on disk: 1.3: rebalance_work Created: Fri Dec 29 06:23:27 2023 Sequence number: 838 Time of last write: Fri Apr 12 21:00:55 2024 Superblock size: 8.70 KiB/1.00 MiB Clean: 0 Devices: 4 Sections: members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade Features: zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 256 KiB metadata_checksum: none crc32c crc64 [xxhash] data_checksum: none crc32c crc64 [xxhash] compression: none background_compression: zstd:15 str_hash: crc32c crc64 [siphash] metadata_target: Device 9324a247-ac99-4af7-a82b-f119b57fbde3 (1) foreground_target: Device 9324a247-ac99-4af7-a82b-f119b57fbde3 (1) background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0 members_v2 (size 528): Device: 0 Label: nvme (1) UUID: 485acf5f-9532-4f5e-89b4-49e700b0be08 Size: 434 GiB read errors: 0 write errors: 0 checksum errors: 5533 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 1778668 Last mount: Fri Apr 12 20:59:07 2024 Last superblock write: 838 State: rw Data allowed: journal,btree,user Has data: journal,btree,user,cached Durability: 1 Discard: 1 Freespace initialized: 1 Device: 1 Label: sata (2) UUID: 9324a247-ac99-4af7-a82b-f119b57fbde3 Size: 466 GiB read errors: 0 write errors: 0 checksum errors: 216396 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 1907760 Last mount: Fri Apr 12 20:59:07 2024 Last superblock write: 838 State: rw Data allowed: journal,btree,user Has data: journal,btree,user,cached Durability: 1 Discard: 1 Freespace initialized: 1 Device: 2 Label: black (4) UUID: 222d1588-f04e-4845-bd8b-eb7d1cac22d1 Size: 932 GiB read errors: 0 write errors: 0 checksum errors: 29 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 3815478 Last mount: Fri Apr 12 20:59:07 2024 Last superblock write: 838 State: rw Data allowed: journal,btree,user Has data: journal,btree,user,cached Durability: 1 Discard: 1 Freespace initialized: 1 Device: 3 Label: green (5) UUID: fe30286b-b190-4e19-9f10-ec4abf57e00a Size: 1.82 TiB read errors: 0 write errors: 0 checksum errors: 68 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 7630916 Last mount: Fri Apr 12 20:59:07 2024 Last superblock write: 838 State: rw Data allowed: journal,btree,user Has data: journal,btree,user,cached Durability: 1 Discard: 1 Freespace initialized: 1 errors (size 264): journal_entry_replicas_not_marked 4 Thu Mar 7 20:52:32 2024 alloc_key_to_missing_lru_entry 141 Sat Mar 2 21:00:35 2024 alloc_key_cached_sectors_wrong 107 Fri Apr 12 21:00:53 2024 bucket_gens_key_wrong 1085535 Wed Jan 31 04:35:13 2024 bucket_gens_hole_wrong 45 Sun Jan 28 08:59:45 2024 need_discard_freespace_key_bad 3627964 Sun Jan 28 09:00:15 2024 ptr_to_missing_backpointer 9271761 Sat Feb 10 07:07:17 2024 ptr_bucket_data_type_mismatch 185 Fri Apr 12 21:00:37 2024 reflink_v_refcount_wrong 508 Fri Apr 12 21:00:55 2024 reflink_p_to_missing_reflink_v 14 Fri Apr 12 21:00:04 2024 inode_dir_wrong_nlink 1 Sat Feb 10 07:09:42 2024 inode_wrong_nlink 4 Sat Feb 10 07:12:09 2024 inode_unreachable 18 Sat Feb 10 07:09:51 2024 deleted_inode_missing 105 Thu Mar 7 01:02:42 2024 inode_bi_parent_wrong 1 Tue Feb 27 21:09:42 2024 unlinked_inode_not_on_deleted_list 105 Thu Mar 7 00:55:47 2024 ```I'm using commit 5639fb38cabaa326b8b664d874a46509d4a60bf2 of bcachefs-tools with a few manual backports from the upstream kernel repository. (Most notably
Fix UAFs of btree_insert_entry arrayas I was fairly consistently running into segfaults caused by said UAF.)Looking at the repair code, if it can't find a matching key, it instead attempts to insert an error key in its place. When this is picked up by the journal replay in a later stage, it. detects that error bkeys aren't allowed in the reflink, eventually resulting in an invalid argument error. It also triggered an emergency ro by default, but another thread suggested that
errors=continuemight be preferred withnochanges, so I added that locally, though It did not progress any further.While the log is from a local version of bcachefs-tools, I was able to reproduce it with the kernel fsck in 6.9-rc3.
Edit: Seems I misread the source code. Also having a journal sequence of -1 seems like it should've been a red flag from the start.
Edit2: Dumping the journal (only the parts that would be replayed, since all of it would be too much) shows no sign of any instances of
146045272or4294967295. Based on that, I'd initially suspect that the reflink btree itself that got corrupted and saved with invalid bkeys, but dumping the reflink btree itself shows a completely normal reflink_v key at0:146045272:0.Edit3: Using the 6.9-rc3 fsck actually gives a slightly different error that looked similar at first glance. Instead it errors from inserting a alloc bkey with a
data_typeoffree. It claims that it got a data type of0that should be9. I'm assuming that0means `free in this context. Also worth noting is that the information given is very different between the kernel and userspace implementations, with the kernel implementation giving information only in the system log rather than its own output, and which seemingly can't be seen at all with the userspace implementation.