fsck: error validating btree node [25de2b00dcca]

After an unclean shutdown, bcachefs fails mount on journal recovery: superblock not marked as containing replicas journal: 1/1 [0], exiting

On fsck:

[    0.000000] Linux version 6.4.2 (nixbld@localhost) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.40) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980
[    0.000000] Command line: BOOT_IMAGE=(hd4,gpt3)/nix/store/igr66d8qmnxfiikid24z3bqd9w5y85za-linux-6.4.2-bcachefs-143f6481/bzImage init=/nix/store/s86yyqqf1yhsz1v1nvp4zvm3csas9yad-nixos-system-wombat-pc-23.11.20230712.2de8efe/init intel_iommu=on iommu=pt zswap.enabled=0 amdgpu.ppfeaturemask=0xffffffff ibt=off systemd.unified_cgroup_hierarchy=0 loglevel=4
...
[   36.362087] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): recovering from unclean shutdown
[   36.362091] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): starting journal read
[   39.105073] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done on device nvme0n1p5, ret 0
[   67.056692] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done on device sda, ret 0
[   82.848914] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done on device sdb, ret 0
[  104.311807] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done on device sdd, ret 0
[  106.774216] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done on device sdc, ret 0
[  106.774240] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): journal read done, replaying entries 21773740-21774177
[  106.776513] superblock not marked as containing replicas journal: 1/1 [0], fixing
[  106.799411] superblock not marked as containing replicas journal: 1/1 [1], fixing
[  106.800958] superblock not marked as containing replicas journal: 1/1 [2], fixing
[  106.802462] superblock not marked as containing replicas journal: 1/1 [3], fixing
[  106.807062] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): Journal keys: 9721 read, 5140 after sorting and compacting
[  107.013173] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): starting alloc read
[  107.174297] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): error validating btree node on nvme0n1p5 at btree bucket_gens level 0/1
[  107.174299]   u64s 12 type btree_ptr_v2 0:23113:0 len 0 ver 0: seq b79889b636ceb207 written 272 min_key 0:22600:1 durability: 2 ptr: 4:69430:0 gen 10 ptr: 0:508334:0 gen 7
[  107.174299]   node offset 0: got wrong btree node (seq 232040b357c13b43 want b79889b636ceb207)
[  107.214128] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): bch2_bucket_gens_read(): error EIO
[  107.214135] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): bch2_fs_recovery(): error EIO
[  107.214136] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): error starting filesystem: EIO
[  107.214137] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): shutting down
[  107.235933] bcachefs (50acd022-b147-4ac2-a47a-36c9f0a239fb): shutdown complete
[  107.236902] bcachefs: bch2_fs_open() ret EIO (-5)

bcachefs 25de2b00dcca9bd801d37efd5e08c15dbf151825 kernel v6.4.2

Kernel options different from Nixos default:

      MODULE_COMPRESS_XZ = lib.mkForce no;
      MODULE_COMPRESS_ZSTD = yes;
      BCACHEFS_FS = yes;
      BCACHEFS_QUOTA = yes;
      BCACHEFS_POSIX_ACL = yes;
      BCACHEFS_LOCK_TIME_STATS = yes;
      BCACHEFS_DEBUG = yes;
      PREEMPT = lib.mkForce yes;
      PREEMPT_VOLUNTARY = lib.mkForce no;
      KALLSYMS = yes;
      KALLSYMS_ALL = yes;
      DEBUG_FS = yes;
      DYNAMIC_FTRACE = yes;
      FTRACE = yes;

notable kernel cmdline: ibt=off

koverstreet / bcachefs

fsck: error validating btree node [25de2b00dcca] #574