list_add corruption. next->prev should be prev

[wmertens]

[47985.623781] ------------[ cut here ]------------
[47985.623784] list_add corruption. next->prev should be prev (ffff910309261548), but was ffff910487cb2d58. (next=ffff910487cb2d58).
[47985.623793] WARNING: CPU: 2 PID: 3920 at lib/list_debug.c:29 __list_add_valid_or_report+0x74/0xc0
[47985.623800] Modules linked in: qrtr ccm rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel ip_set_hash_net af_packet cmac algif_hash algif_skcipher af_alg bnep msr xt_set ip_set_hash_ipport ip_set xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel uinput nls_iso8859_1 nls_cp437 vfat fat snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci nvidia_drm(PO) snd_sof_xtensa_dsp mt7921e snd_sof mt7921_common nvidia_modeset(PO) mt792x_lib snd_sof_utils mt76_connac_lib snd_pci_ps snd_amd_sdw_acpi mt76 snd_hda_codec_realtek soundwire_amd soundwire_generic_allocation nvidia_uvm(PO) soundwire_bus snd_hda_codec_generic mac80211 snd_hda_scodec_component snd_soc_core snd_hda_codec_hdmi joydev mousedev snd_compress snd_hda_intel
[47985.623896]  ac97_bus snd_pcm_dmaengine snd_rpl_pci_acp6x snd_intel_dspcfg snd_acp_pci snd_intel_sdw_acpi edac_mce_amd snd_acp_legacy_common snd_hda_codec edac_core btusb snd_pci_acp6x intel_rapl_msr hid_sensor_als hid_sensor_trigger amd_atl btrtl snd_hda_core uvcvideo intel_rapl_common industrialio_triggered_buffer snd_pci_acp5x snd_aloop kfifo_buf btintel crc32_pclmul snd_hwdep hid_sensor_iio_common btbcm videobuf2_vmalloc snd_rn_pci_acp3x polyval_clmulni industrialio hid_sensor_custom snd_pcm ucsi_acpi snd_acp_config sp5100_tco btmtk polyval_generic uvc cfg80211 bluetooth snd_soc_acpi typec_ucsi watchdog hid_sensor_hub snd_timer ideapad_laptop gf128mul videobuf2_memops hid_multitouch hid_generic nvidia(PO) ghash_clmulni_intel ecdh_generic sha512_ssse3 sha1_ssse3 aesni_intel videobuf2_v4l2 ecc snd crypto_simd cryptd crc16 videobuf2_common platform_profile rapl wmi_bmof typec tpm_crb tiny_power_button sparse_keymap snd_pci_acp3x soundcore k10temp rfkill i2c_piix4 libarc4 thermal roles v4l2loopback(O) battery evdev
[47985.623999]  button i2c_hid_acpi i2c_hid tpm_tis hid tpm_tis_core input_leds videodev amd_pmc ac mac_hid serio_raw mc led_class loop cpufreq_ondemand tun tap macvlan bridge nfsd stp llc kvm_amd auth_rpcgss ccp nfs_acl lockd grace kvm sunrpc fuse efi_pstore configfs nfnetlink zram efivarfs tpm rng_core dmi_sysfs ip_tables x_tables autofs4 bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq xhci_pci nvme atkbd xhci_pci_renesas thunderbolt libps2 nvme_core vivaldi_fmap xhci_hcd nvme_auth t10_pi crc32c_intel sha256_ssse3 crc64_rocksoft crc_t10dif crct10dif_generic i8042 crct10dif_pclmul crc64 crct10dif_common rtc_cmos serio dm_mod dax amdgpu video wmi backlight amdxcp i2c_algo_bit drm_ttm_helper ttm drm_exec gpu_sched drm_suballoc_helper drm_buddy drm_display_helper firmware_class cec
[47985.624113] CPU: 2 PID: 3920 Comm: ThreadPoolForeg Tainted: P      D W  O       6.9.4 #1-NixOS
[47985.624118] Hardware name: LENOVO 82Y8/LNVNB161216, BIOS MLCN28WW 01/22/2024
[47985.624121] RIP: 0010:__list_add_valid_or_report+0x74/0xc0
[47985.624127] Code: e9 bc 65 8e 00 90 48 c7 c7 58 35 15 95 e8 14 8c a9 ff 90 0f 0b 90 90 eb e5 90 48 c7 c7 80 35 15 95 48 89 c1 e8 fd 8b a9 ff 90 <0f> 0b 90 90 eb ce 90 48 c7 c7 d0 35 15 95 48 89 d1 48 89 c6 4c 89
[47985.624131] RSP: 0018:ffff9a5cc86f39d0 EFLAGS: 00010282
[47985.624136] RAX: 0000000000000000 RBX: ffff91020274c9c0 RCX: 0000000000000027
[47985.624139] RDX: ffff91060fb21848 RSI: 0000000000000001 RDI: ffff91060fb21840
[47985.624143] RBP: ffff91020274cad8 R08: 0000000000000000 R09: 0000000000000003
[47985.624146] R10: ffff9a5cc86f3878 R11: ffffffff9590f6c0 R12: ffff910309261000
[47985.624149] R13: ffff910487cb2d58 R14: ffff910309261548 R15: ffffffffc17240d0
[47985.624153] FS:  00007f6e71c006c0(0000) GS:ffff91060fb00000(0000) knlGS:0000000000000000
[47985.624157] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[47985.624161] CR2: 00007eff7c6f7960 CR3: 0000000233c0a000 CR4: 0000000000f50ef0
[47985.624164] PKRU: 55555554
[47985.624167] Call Trace:
[47985.624170]  <TASK>
[47985.624173]  ? __warn+0x80/0x120
[47985.624178]  ? __list_add_valid_or_report+0x74/0xc0
[47985.624185]  ? report_bug+0x164/0x190
[47985.624191]  ? handle_bug+0x3d/0x80
[47985.624197]  ? exc_invalid_op+0x17/0x70
[47985.624203]  ? asm_exc_invalid_op+0x1a/0x20
[47985.624208]  ? __pfx_bch2_iget5_set+0x10/0x10 [bcachefs]
[47985.624290]  ? __list_add_valid_or_report+0x74/0xc0
[47985.624296]  inode_insert5+0x14e/0x1f0
[47985.624306]  bch2_inode_insert+0xad/0x170 [bcachefs]
[47985.624391]  ? bch2_inode_update_after_write+0x79/0x180 [bcachefs]
[47985.624469]  __bch2_create+0x4ca/0x5c0 [bcachefs]
[47985.624562]  ? bch2_create+0x2a/0x60 [bcachefs]
[47985.624638]  bch2_create+0x2a/0x60 [bcachefs]
[47985.624716]  path_openat+0xe92/0x1160
[47985.624727]  do_filp_open+0xc4/0x170
[47985.624742]  do_sys_openat2+0xab/0xe0
[47985.624750]  __x64_sys_openat+0x57/0xa0
[47985.624756]  do_syscall_64+0xb8/0x200
[47985.624762]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[47985.624769] RIP: 0033:0x7f6e906fe330
[47985.624783] Code: 48 89 44 24 20 75 94 44 89 54 24 0c e8 09 e2 f8 ff 44 8b 54 24 0c 89 da 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 89 44 24 0c e8 5c e2 f8 ff 8b 44
[47985.624788] RSP: 002b:00007f6e71bfd500 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[47985.624794] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007f6e906fe330
[47985.624798] RDX: 0000000000000241 RSI: 000005080ea78d00 RDI: 00000000ffffff9c
[47985.624801] RBP: 000005080ea78d00 R08: 0000000000000000 R09: 0000000000000004
[47985.624805] R10: 0000000000000180 R11: 0000000000000293 R12: 0000000000000241
[47985.624808] R13: 0000000000000241 R14: 00007f6e71bfd680 R15: 0000000000000048
[47985.624817]  </TASK>
[47985.624820] ---[ end trace 0000000000000000 ]---

This pops up quite often.

When I try to online repair:

root@wmertens-nixos:~]# bcachefs fsck /dev/disk/by-partlabel/root 
bcachefs (nvme0n1p5): check_alloc_info... done
bcachefs (nvme0n1p5): check_lrus... done
bcachefs (nvme0n1p5): check_btree_backpointers... done
bcachefs (nvme0n1p5): check_backpointers_to_extents... done
bcachefs (nvme0n1p5): check_extents_to_backpointers... done
bcachefs (nvme0n1p5): check_alloc_to_lru_refs... done
bcachefs (nvme0n1p5): check_snapshot_trees... done
bcachefs (nvme0n1p5): check_snapshots... done
bcachefs (nvme0n1p5): check_subvols... done
bcachefs (nvme0n1p5): check_subvol_children... done
bcachefs (nvme0n1p5): delete_dead_snapshots... done
bcachefs (nvme0n1p5): check_root... done
bcachefs (nvme0n1p5): check_subvolume_structure... done
bcachefs (nvme0n1p5): check_directory_structure...unreachable inode
u64s 17 type inode_v3 0:4628:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6337076
  bi_size=1037
  bi_sectors=3
  bi_version=0
  bi_atime=5291430689489397
  bi_ctime=5291596446396918
  bi_mtime=16733396170247297669
  bi_otime=324226083446
  bi_uid=0
  bi_gid=0
  bi_nlink=0
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
: fix? (y,n, or Y,N for all errors of this type) Y
unreachable inode
u64s 17 type inode_v3 0:4910:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6243706
  bi_size=13926
  bi_sectors=28
  bi_version=0
  bi_atime=5205699583457295
  bi_ctime=5205699220385128
  bi_mtime=16733396170247297669
  bi_otime=324325079290
  bi_uid=0
  bi_gid=0
  bi_nlink=0
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
, fixing
unreachable inode
u64s 17 type inode_v3 0:598054:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6391351
  bi_size=15
  bi_sectors=1
  bi_version=0
  bi_atime=8883949099173
  bi_ctime=5353742152125430
  bi_mtime=16733396170247297669
  bi_otime=8883220134903
  bi_uid=0
  bi_gid=0
  bi_nlink=7
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
, fixing
bcachefs (nvme0n1p5): check_path(): error EEXIST
bcachefs (nvme0n1p5): bch2_check_directory_structure(): error EEXIST
bcachefs (nvme0n1p5): bch2_fsck_online_thread_fn(): error EEXIST

[koverstreet]

What version? the EEXIST makes me think you might be running an old one

[wmertens]

Apologies, should have added that from the start, looks like 1.7:

$ bcachefs version
1.7.0-unstable-2024-05-09

$ sudo bcachefs show-super /dev/disk/by-partlabel/root
Device:                                     (unknown device)
External UUID:                             82ace8da-b608-450f-a397-f1b30c838560
Internal UUID:                             98c4b18b-c7de-4491-a600-d81d09cee632
Magic number:                              c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                              0
Label:
Version:        1.7: mi_btree_bitmap
Version upgrade complete:       1.7: mi_btree_bitmap
Oldest version on disk:                    1.4: member_seq
Created:                                   Wed Apr 17 11:58:24 2024
Sequence number:                           258
Time of last write:                        Wed Jun 19 14:43:33 2024
Superblock size:                           5.79 KiB/1.00 MiB
Clean:                                     0
Devices:                                   1
Sections:                                  members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                  zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                           alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                              512 B
  btree_node_size:                         256 KiB
  errors:                                  continue [ro] panic 
  metadata_replicas:                       1
  data_replicas:                           1
  metadata_replicas_required:              1
  data_replicas_required:                  1
  encoded_extent_max:                      64.0 KiB
  metadata_checksum:                       none [crc32c] crc64 xxhash 
  data_checksum:                           none [crc32c] crc64 xxhash 
  compression:                             zstd
  background_compression:                  none
  str_hash:                                crc32c crc64 [siphash] 
  metadata_target:                         none
  foreground_target:                       none
  background_target:                       none
  promote_target:                          none
  erasure_code:                            0
  inodes_32bit:                            1
  shard_inode_numbers:                     1
  inodes_use_key_cache:                    1
  gc_reserve_percent:                      8
  gc_reserve_bytes:                        0 B
  root_reserve_percent:                    0
  wide_macs:                               0
  acl:                                     1
  usrquota:                                0
  grpquota:                                0
  prjquota:                                0
  journal_flush_delay:                     1000
  journal_flush_disabled:                  0
  journal_reclaim_delay:                   100
  journal_transaction_names:               1
  version_upgrade:                         [compatible] incompatible none 
  nocow:                                   0

members_v2 (size 152):
Device:                                    0
  Label:                                   root (0)
  UUID:                                    47c12cb4-d0d7-4220-b545-35c26f85eb40
  Size:                                    837 GiB
  read errors:                             0
  write errors:                            0
  checksum errors:                         0
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             512 KiB
  First bucket:                            0
  Buckets:                                 1714424
  Last mount:                              Wed Jun 19 14:43:31 2024
  Last superblock write:                   258
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user
  Btree allocated bitmap blocksize:        32.0 MiB
  Btree allocated bitmap:                  0000000000000000000000001111111111111111111111111111111111111111
  Durability:                              1
  Discard:                                 0
  Freespace initialized:                   1

errors (size 40):
inode_multiple_links_but_nlink_0            236             Mon May 27 08:03:13 2024
inode_wrong_nlink                           192             Mon May 27 08:04:01 2024

$ uname -a
Linux wmertens-nixos 6.9.4 #1-NixOS SMP PREEMPT_DYNAMIC Wed Jun 12 09:39:59 UTC 2024 x86_64 GNU/Linux

[koverstreet]

6.9.4 is recent enough, something odd is going on.

Could you send me a metadata dump? Jump on IRC (irc.oftc.net#bcache) and send it to me via magic wormhole.

Also, from your kernel source tree, ./scripts/faddr2line vmlinux inode_insert5+0x14e/0x1f0

[wmertens]

@koverstreet that needs to happen offline, right? How big should I expect the metadata to be? I'll have to do it from a rescue image.

[koverstreet]

correct bcachefs fs usage should give you an idea - look at the amount of metadata you have

[koverstreet]

Update on this? I want to get your fs back up and running :)

Is the metadata dump going to be practical?

[koverstreet]

Turns out that bare EEXIST is from us; I'd fixed it in master, but that wasn't in 6.9.

I should be able to fix that today.

[wmertens]

ah - so everything actually works and I have no visible corruption, only the messages and the dangling inodes.

So if you no longer need the metadata that's good news because it is quite an ordeal for me. I also need to figure out how to get faddr2line from nixpkgs.

[koverstreet]

fsck is still giving you EEXIST when it tries to repair?