search

koverstreet / bcachefs

Other
695 stars 72 forks source link

list_add corruption. next->prev should be prev #693

Closed wmertens closed 3 months ago

wmertens commented 4 months ago 
[47985.623781] ------------[ cut here ]------------
[47985.623784] list_add corruption. next->prev should be prev (ffff910309261548), but was ffff910487cb2d58. (next=ffff910487cb2d58).
[47985.623793] WARNING: CPU: 2 PID: 3920 at lib/list_debug.c:29 __list_add_valid_or_report+0x74/0xc0
[47985.623800] Modules linked in: qrtr ccm rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel ip_set_hash_net af_packet cmac algif_hash algif_skcipher af_alg bnep msr xt_set ip_set_hash_ipport ip_set xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel uinput nls_iso8859_1 nls_cp437 vfat fat snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci nvidia_drm(PO) snd_sof_xtensa_dsp mt7921e snd_sof mt7921_common nvidia_modeset(PO) mt792x_lib snd_sof_utils mt76_connac_lib snd_pci_ps snd_amd_sdw_acpi mt76 snd_hda_codec_realtek soundwire_amd soundwire_generic_allocation nvidia_uvm(PO) soundwire_bus snd_hda_codec_generic mac80211 snd_hda_scodec_component snd_soc_core snd_hda_codec_hdmi joydev mousedev snd_compress snd_hda_intel
[47985.623896]  ac97_bus snd_pcm_dmaengine snd_rpl_pci_acp6x snd_intel_dspcfg snd_acp_pci snd_intel_sdw_acpi edac_mce_amd snd_acp_legacy_common snd_hda_codec edac_core btusb snd_pci_acp6x intel_rapl_msr hid_sensor_als hid_sensor_trigger amd_atl btrtl snd_hda_core uvcvideo intel_rapl_common industrialio_triggered_buffer snd_pci_acp5x snd_aloop kfifo_buf btintel crc32_pclmul snd_hwdep hid_sensor_iio_common btbcm videobuf2_vmalloc snd_rn_pci_acp3x polyval_clmulni industrialio hid_sensor_custom snd_pcm ucsi_acpi snd_acp_config sp5100_tco btmtk polyval_generic uvc cfg80211 bluetooth snd_soc_acpi typec_ucsi watchdog hid_sensor_hub snd_timer ideapad_laptop gf128mul videobuf2_memops hid_multitouch hid_generic nvidia(PO) ghash_clmulni_intel ecdh_generic sha512_ssse3 sha1_ssse3 aesni_intel videobuf2_v4l2 ecc snd crypto_simd cryptd crc16 videobuf2_common platform_profile rapl wmi_bmof typec tpm_crb tiny_power_button sparse_keymap snd_pci_acp3x soundcore k10temp rfkill i2c_piix4 libarc4 thermal roles v4l2loopback(O) battery evdev
[47985.623999]  button i2c_hid_acpi i2c_hid tpm_tis hid tpm_tis_core input_leds videodev amd_pmc ac mac_hid serio_raw mc led_class loop cpufreq_ondemand tun tap macvlan bridge nfsd stp llc kvm_amd auth_rpcgss ccp nfs_acl lockd grace kvm sunrpc fuse efi_pstore configfs nfnetlink zram efivarfs tpm rng_core dmi_sysfs ip_tables x_tables autofs4 bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq xhci_pci nvme atkbd xhci_pci_renesas thunderbolt libps2 nvme_core vivaldi_fmap xhci_hcd nvme_auth t10_pi crc32c_intel sha256_ssse3 crc64_rocksoft crc_t10dif crct10dif_generic i8042 crct10dif_pclmul crc64 crct10dif_common rtc_cmos serio dm_mod dax amdgpu video wmi backlight amdxcp i2c_algo_bit drm_ttm_helper ttm drm_exec gpu_sched drm_suballoc_helper drm_buddy drm_display_helper firmware_class cec
[47985.624113] CPU: 2 PID: 3920 Comm: ThreadPoolForeg Tainted: P      D W  O       6.9.4 #1-NixOS
[47985.624118] Hardware name: LENOVO 82Y8/LNVNB161216, BIOS MLCN28WW 01/22/2024
[47985.624121] RIP: 0010:__list_add_valid_or_report+0x74/0xc0
[47985.624127] Code: e9 bc 65 8e 00 90 48 c7 c7 58 35 15 95 e8 14 8c a9 ff 90 0f 0b 90 90 eb e5 90 48 c7 c7 80 35 15 95 48 89 c1 e8 fd 8b a9 ff 90 <0f> 0b 90 90 eb ce 90 48 c7 c7 d0 35 15 95 48 89 d1 48 89 c6 4c 89
[47985.624131] RSP: 0018:ffff9a5cc86f39d0 EFLAGS: 00010282
[47985.624136] RAX: 0000000000000000 RBX: ffff91020274c9c0 RCX: 0000000000000027
[47985.624139] RDX: ffff91060fb21848 RSI: 0000000000000001 RDI: ffff91060fb21840
[47985.624143] RBP: ffff91020274cad8 R08: 0000000000000000 R09: 0000000000000003
[47985.624146] R10: ffff9a5cc86f3878 R11: ffffffff9590f6c0 R12: ffff910309261000
[47985.624149] R13: ffff910487cb2d58 R14: ffff910309261548 R15: ffffffffc17240d0
[47985.624153] FS:  00007f6e71c006c0(0000) GS:ffff91060fb00000(0000) knlGS:0000000000000000
[47985.624157] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[47985.624161] CR2: 00007eff7c6f7960 CR3: 0000000233c0a000 CR4: 0000000000f50ef0
[47985.624164] PKRU: 55555554
[47985.624167] Call Trace:
[47985.624170]  <TASK>
[47985.624173]  ? __warn+0x80/0x120
[47985.624178]  ? __list_add_valid_or_report+0x74/0xc0
[47985.624185]  ? report_bug+0x164/0x190
[47985.624191]  ? handle_bug+0x3d/0x80
[47985.624197]  ? exc_invalid_op+0x17/0x70
[47985.624203]  ? asm_exc_invalid_op+0x1a/0x20
[47985.624208]  ? __pfx_bch2_iget5_set+0x10/0x10 [bcachefs]
[47985.624290]  ? __list_add_valid_or_report+0x74/0xc0
[47985.624296]  inode_insert5+0x14e/0x1f0
[47985.624306]  bch2_inode_insert+0xad/0x170 [bcachefs]
[47985.624391]  ? bch2_inode_update_after_write+0x79/0x180 [bcachefs]
[47985.624469]  __bch2_create+0x4ca/0x5c0 [bcachefs]
[47985.624562]  ? bch2_create+0x2a/0x60 [bcachefs]
[47985.624638]  bch2_create+0x2a/0x60 [bcachefs]
[47985.624716]  path_openat+0xe92/0x1160
[47985.624727]  do_filp_open+0xc4/0x170
[47985.624742]  do_sys_openat2+0xab/0xe0
[47985.624750]  __x64_sys_openat+0x57/0xa0
[47985.624756]  do_syscall_64+0xb8/0x200
[47985.624762]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[47985.624769] RIP: 0033:0x7f6e906fe330
[47985.624783] Code: 48 89 44 24 20 75 94 44 89 54 24 0c e8 09 e2 f8 ff 44 8b 54 24 0c 89 da 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 89 44 24 0c e8 5c e2 f8 ff 8b 44
[47985.624788] RSP: 002b:00007f6e71bfd500 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[47985.624794] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007f6e906fe330
[47985.624798] RDX: 0000000000000241 RSI: 000005080ea78d00 RDI: 00000000ffffff9c
[47985.624801] RBP: 000005080ea78d00 R08: 0000000000000000 R09: 0000000000000004
[47985.624805] R10: 0000000000000180 R11: 0000000000000293 R12: 0000000000000241
[47985.624808] R13: 0000000000000241 R14: 00007f6e71bfd680 R15: 0000000000000048
[47985.624817]  </TASK>
[47985.624820] ---[ end trace 0000000000000000 ]---

This pops up quite often.

When I try to online repair:

root@wmertens-nixos:~]# bcachefs fsck /dev/disk/by-partlabel/root 
bcachefs (nvme0n1p5): check_alloc_info... done
bcachefs (nvme0n1p5): check_lrus... done
bcachefs (nvme0n1p5): check_btree_backpointers... done
bcachefs (nvme0n1p5): check_backpointers_to_extents... done
bcachefs (nvme0n1p5): check_extents_to_backpointers... done
bcachefs (nvme0n1p5): check_alloc_to_lru_refs... done
bcachefs (nvme0n1p5): check_snapshot_trees... done
bcachefs (nvme0n1p5): check_snapshots... done
bcachefs (nvme0n1p5): check_subvols... done
bcachefs (nvme0n1p5): check_subvol_children... done
bcachefs (nvme0n1p5): delete_dead_snapshots... done
bcachefs (nvme0n1p5): check_root... done
bcachefs (nvme0n1p5): check_subvolume_structure... done
bcachefs (nvme0n1p5): check_directory_structure...unreachable inode
u64s 17 type inode_v3 0:4628:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6337076
  bi_size=1037
  bi_sectors=3
  bi_version=0
  bi_atime=5291430689489397
  bi_ctime=5291596446396918
  bi_mtime=16733396170247297669
  bi_otime=324226083446
  bi_uid=0
  bi_gid=0
  bi_nlink=0
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
: fix? (y,n, or Y,N for all errors of this type) Y
unreachable inode
u64s 17 type inode_v3 0:4910:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6243706
  bi_size=13926
  bi_sectors=28
  bi_version=0
  bi_atime=5205699583457295
  bi_ctime=5205699220385128
  bi_mtime=16733396170247297669
  bi_otime=324325079290
  bi_uid=0
  bi_gid=0
  bi_nlink=0
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
, fixing
unreachable inode
u64s 17 type inode_v3 0:598054:U32_MAX len 0 ver 0:   mode=100444
  flags= (b300000)
  journal_seq=6391351
  bi_size=15
  bi_sectors=1
  bi_version=0
  bi_atime=8883949099173
  bi_ctime=5353742152125430
  bi_mtime=16733396170247297669
  bi_otime=8883220134903
  bi_uid=0
  bi_gid=0
  bi_nlink=7
  bi_generation=0
  bi_dev=0
  bi_data_checksum=0
  bi_compression=4
  bi_project=0
  bi_background_compression=0
  bi_data_replicas=0
  bi_promote_target=0
  bi_foreground_target=0
  bi_background_target=0
  bi_erasure_code=0
  bi_fields_set=0
  bi_dir=0
  bi_dir_offset=0
  bi_subvol=0
  bi_parent_subvol=0
  bi_nocow=0
, fixing
bcachefs (nvme0n1p5): check_path(): error EEXIST
bcachefs (nvme0n1p5): bch2_check_directory_structure(): error EEXIST
bcachefs (nvme0n1p5): bch2_fsck_online_thread_fn(): error EEXIST
koverstreet commented 4 months ago

What version? the EEXIST makes me think you might be running an old one

wmertens commented 4 months ago

Apologies, should have added that from the start, looks like 1.7:

$ bcachefs version
1.7.0-unstable-2024-05-09

$ sudo bcachefs show-super /dev/disk/by-partlabel/root
Device:                                     (unknown device)
External UUID:                             82ace8da-b608-450f-a397-f1b30c838560
Internal UUID:                             98c4b18b-c7de-4491-a600-d81d09cee632
Magic number:                              c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                              0
Label:
Version:        1.7: mi_btree_bitmap
Version upgrade complete:       1.7: mi_btree_bitmap
Oldest version on disk:                    1.4: member_seq
Created:                                   Wed Apr 17 11:58:24 2024
Sequence number:                           258
Time of last write:                        Wed Jun 19 14:43:33 2024
Superblock size:                           5.79 KiB/1.00 MiB
Clean:                                     0
Devices:                                   1
Sections:                                  members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                  zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                           alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                              512 B
  btree_node_size:                         256 KiB
  errors:                                  continue [ro] panic 
  metadata_replicas:                       1
  data_replicas:                           1
  metadata_replicas_required:              1
  data_replicas_required:                  1
  encoded_extent_max:                      64.0 KiB
  metadata_checksum:                       none [crc32c] crc64 xxhash 
  data_checksum:                           none [crc32c] crc64 xxhash 
  compression:                             zstd
  background_compression:                  none
  str_hash:                                crc32c crc64 [siphash] 
  metadata_target:                         none
  foreground_target:                       none
  background_target:                       none
  promote_target:                          none
  erasure_code:                            0
  inodes_32bit:                            1
  shard_inode_numbers:                     1
  inodes_use_key_cache:                    1
  gc_reserve_percent:                      8
  gc_reserve_bytes:                        0 B
  root_reserve_percent:                    0
  wide_macs:                               0
  acl:                                     1
  usrquota:                                0
  grpquota:                                0
  prjquota:                                0
  journal_flush_delay:                     1000
  journal_flush_disabled:                  0
  journal_reclaim_delay:                   100
  journal_transaction_names:               1
  version_upgrade:                         [compatible] incompatible none 
  nocow:                                   0

members_v2 (size 152):
Device:                                    0
  Label:                                   root (0)
  UUID:                                    47c12cb4-d0d7-4220-b545-35c26f85eb40
  Size:                                    837 GiB
  read errors:                             0
  write errors:                            0
  checksum errors:                         0
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             512 KiB
  First bucket:                            0
  Buckets:                                 1714424
  Last mount:                              Wed Jun 19 14:43:31 2024
  Last superblock write:                   258
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user
  Btree allocated bitmap blocksize:        32.0 MiB
  Btree allocated bitmap:                  0000000000000000000000001111111111111111111111111111111111111111
  Durability:                              1
  Discard:                                 0
  Freespace initialized:                   1

errors (size 40):
inode_multiple_links_but_nlink_0            236             Mon May 27 08:03:13 2024
inode_wrong_nlink                           192             Mon May 27 08:04:01 2024

$ uname -a
Linux wmertens-nixos 6.9.4 #1-NixOS SMP PREEMPT_DYNAMIC Wed Jun 12 09:39:59 UTC 2024 x86_64 GNU/Linux
koverstreet commented 4 months ago

6.9.4 is recent enough, something odd is going on.

Could you send me a metadata dump? Jump on IRC (irc.oftc.net#bcache) and send it to me via magic wormhole.

Also, from your kernel source tree, ./scripts/faddr2line vmlinux inode_insert5+0x14e/0x1f0

wmertens commented 4 months ago

@koverstreet that needs to happen offline, right? How big should I expect the metadata to be? I'll have to do it from a rescue image.

koverstreet commented 4 months ago

correct bcachefs fs usage should give you an idea - look at the amount of metadata you have

koverstreet commented 4 months ago

Update on this? I want to get your fs back up and running :)

Is the metadata dump going to be practical?

koverstreet commented 4 months ago

Turns out that bare EEXIST is from us; I'd fixed it in master, but that wasn't in 6.9.

I should be able to fix that today.

wmertens commented 4 months ago

ah - so everything actually works and I have no visible corruption, only the messages and the dangling inodes.

So if you no longer need the metadata that's good news because it is quite an ordeal for me. I also need to figure out how to get faddr2line from nixpkgs.

koverstreet commented 4 months ago

fsck is still giving you EEXIST when it tries to repair?

wucke13 commented 4 months ago

Just chiming in here, this also happens on my NixOS + bcachefs system. The output of dmesg is full of this, and Disk IO is unusually slow, but the thing works in general. No visible file loss so far. Running Linux zorn 6.9.6 #1-NixOS SMP PREEMPT_DYNAMIC. I will reboot later and put an update if that fixes log spam im dmesg.

koverstreet commented 4 months ago

Can one of you check if it's still happening on 6.10-rc?

wucke13 commented 4 months ago

So I rebooted the machine, but it got in a kernel panick. I used the HW reset switch to reboot, and ran fsck.bcachefs -p -y -f -k -v /dev/nvme0n1p2. This seems to have run quite uneventful, log below. The machine is now up an running again. I got a couple of other bcachefs related warnings, but IO peformance seems good (much bettern than yesterday), and this only a handful of warnings vs the 50 or so kernel panicks per second yesterday.

I will try to go for 6.10-rc and report back, if I do so.

bcachefs (nvme0n1p2): mounting version 1.7: mi_btree_bitmap opts=ro,degraded,verbose,fsck,fix_errors=yes,read_only
bcachefs (nvme0n1p2): recovering from unclean shutdown
bcachefs (nvme0n1p2): starting journal read
bcachefs (nvme0n1p2): journal read done on device nvme0n1p2, ret 0
bcachefs (nvme0n1p2): journal read done, replaying entries 50236341-50236341
bcachefs (nvme0n1p2): Journal keys: 0 read, 0 after sorting and compacting
bcachefs (nvme0n1p2): alloc_read... done
bcachefs (nvme0n1p2): stripes_read... done
bcachefs (nvme0n1p2): snapshots_read... done
bcachefs (nvme0n1p2): check_allocations... done
bcachefs (nvme0n1p2): going read-write
bcachefs (nvme0n1p2): journal_replay... done
bcachefs (nvme0n1p2): check_alloc_info... done
bcachefs (nvme0n1p2): check_lrus... done
bcachefs (nvme0n1p2): check_btree_backpointers... done
bcachefs (nvme0n1p2): check_backpointers_to_extents... done
bcachefs (nvme0n1p2): check_extents_to_backpointers... done
bcachefs (nvme0n1p2): check_alloc_to_lru_refs... done
bcachefs (nvme0n1p2): check_snapshot_trees... done
bcachefs (nvme0n1p2): check_snapshots... done
bcachefs (nvme0n1p2): check_subvols... done
bcachefs (nvme0n1p2): check_subvol_children... done
bcachefs (nvme0n1p2): delete_dead_snapshots... done
bcachefs (nvme0n1p2): check_inodes... done
bcachefs (nvme0n1p2): check_extents... done
bcachefs (nvme0n1p2): check_indirect_extents... done
bcachefs (nvme0n1p2): check_dirents... done
bcachefs (nvme0n1p2): check_xattrs... done
bcachefs (nvme0n1p2): check_root... done
bcachefs (nvme0n1p2): check_subvolume_structure... done
bcachefs (nvme0n1p2): check_directory_structure... done
bcachefs (nvme0n1p2): check_nlinks... done
bcachefs (nvme0n1p2): resume_logged_ops... done
bcachefs (nvme0n1p2): delete_dead_inodes... done
bcachefs (nvme0n1p2): going read-only
bcachefs (nvme0n1p2): finished waiting for writes to stop
bcachefs (nvme0n1p2): flushing journal and stopping allocators, journal seq 50236704
bcachefs (nvme0n1p2): flushing journal and stopping allocators complete, journal seq 50236713
bcachefs (nvme0n1p2): shutdown complete, journal seq 50236714
bcachefs (nvme0n1p2): marking filesystem clean
bcachefs (nvme0n1p2): done starting filesystem
bcachefs (nvme0n1p2): shutting down
bcachefs (nvme0n1p2): shutdown complete
[33102.639053] ------------[ cut here ]------------
[33102.639057] btree trans held srcu lock (delaying memory reclaim) for 10 seconds
[33102.639108] WARNING: CPU: 7 PID: 1956 at fs/bcachefs/btree_iter.c:2871 bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs]
[33102.639169] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class
[33102.639263]  nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic
[33102.639370]  dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4
[33102.639385] CPU: 7 PID: 1956 Comm: zerotier-one Tainted: G           O       6.9.8 #1-NixOS
[33102.639389] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[33102.639391] RIP: 0010:bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs]
[33102.639442] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 c6 fc 87 d6 90 <0f> 0b 90 90 e9 5f ff ff ff 90 0f 0b 90 e9 6c ff ff ff 0f 1f 00 90
[33102.639445] RSP: 0018:ffffb2a40163f9b8 EFLAGS: 00010282
[33102.639449] RAX: 0000000000000000 RBX: ffff931e07f34000 RCX: 0000000000000027
[33102.639451] RDX: ffff9320ceba1848 RSI: 0000000000000001 RDI: ffff9320ceba1840
[33102.639453] RBP: ffff931dd50c0000 R08: 0000000000000000 R09: 0000000000000003
[33102.639456] R10: ffffb2a40163f860 R11: ffffffff98f3a128 R12: ffffb2a40163fa80
[33102.639458] R13: ffff931e07f34000 R14: ffff931e07f34000 R15: ffff9320cc55e508
[33102.639460] FS:  00007fc0892006c0(0000) GS:ffff9320ceb80000(0000) knlGS:0000000000000000
[33102.639463] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33102.639466] CR2: 00007f5f74bbcce0 CR3: 00000001bb260000 CR4: 00000000003506f0
[33102.639469] Call Trace:
[33102.639472]  <TASK>
[33102.639477]  ? __warn+0x80/0x120
[33102.639482]  ? bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs]
[33102.639534]  ? report_bug+0x164/0x190
[33102.639541]  ? handle_bug+0x3d/0x80
[33102.639547]  ? exc_invalid_op+0x17/0x70
[33102.639551]  ? asm_exc_invalid_op+0x1a/0x20
[33102.639558]  ? bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs]
[33102.639610]  ? bch2_trans_srcu_unlock+0x11a/0x130 [bcachefs]
[33102.639659]  ? bch2_trans_begin+0xf8/0x600 [bcachefs]
[33102.639707]  bch2_trans_begin+0x5a5/0x600 [bcachefs]
[33102.639759]  bch2_logged_op_start+0x1d/0xa0 [bcachefs]
[33102.639838]  bch2_truncate+0xa6/0x110 [bcachefs]
[33102.639904]  bchfs_truncate+0x274/0x3c0 [bcachefs]
[33102.639969]  notify_change+0x1f5/0x4c0
[33102.639977]  ? do_truncate+0x98/0xf0
[33102.639982]  ? srso_return_thunk+0x5/0x5f
[33102.639986]  do_truncate+0x98/0xf0
[33102.639993]  path_openat+0xf96/0x1150
[33102.640001]  do_filp_open+0xc4/0x170
[33102.640011]  do_sys_openat2+0xab/0xe0
[33102.640014]  ? srso_return_thunk+0x5/0x5f
[33102.640019]  __x64_sys_openat+0x57/0xa0
[33102.640023]  do_syscall_64+0xb8/0x200
[33102.640028]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[33102.640031] RIP: 0033:0x7fc0894a1330
[33102.640066] Code: 48 89 44 24 20 75 94 44 89 54 24 0c e8 09 e2 f8 ff 44 8b 54 24 0c 89 da 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 89 44 24 0c e8 5c e2 f8 ff 8b 44
[33102.640069] RSP: 002b:00007fc0891ff820 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[33102.640072] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007fc0894a1330
[33102.640074] RDX: 0000000000000241 RSI: 0000561011357590 RDI: 00000000ffffff9c
[33102.640076] RBP: 0000561011357590 R08: 0000000000000000 R09: 0000000000000001
[33102.640078] R10: 00000000000001b6 R11: 0000000000000293 R12: 00007fc089864902
[33102.640080] R13: 00007fc089864901 R14: 0000000000000001 R15: 00007fc0891ffa10
[33102.640086]  </TASK>
[33102.640088] ---[ end trace 0000000000000000 ]---
[33119.136907] ------------[ cut here ]------------
[33119.136915] btree trans held srcu lock (delaying memory reclaim) for 29 seconds
[33119.136960] WARNING: CPU: 13 PID: 9254 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
[33119.137019] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class
[33119.137110]  nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic
[33119.137215]  dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4
[33119.137230] CPU: 13 PID: 9254 Comm: ForkJoinPool-1- Tainted: G        W  O       6.9.8 #1-NixOS
[33119.137234] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[33119.137236] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
[33119.137296] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
[33119.137299] RSP: 0018:ffffb2a400d4b988 EFLAGS: 00010282
[33119.137302] RAX: 0000000000000000 RBX: ffff931f99213a00 RCX: 0000000000000027
[33119.137305] RDX: ffff9320ceea1848 RSI: 0000000000000001 RDI: ffff9320ceea1840
[33119.137307] RBP: ffff931e5c340000 R08: 0000000000000000 R09: 0000000000000003
[33119.137310] R10: ffffb2a400d4b830 R11: ffffffff98f3a128 R12: ffff931e5c340000
[33119.137312] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000
[33119.137314] FS:  00007fffb36006c0(0000) GS:ffff9320cee80000(0000) knlGS:0000000000000000
[33119.137320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33119.137322] CR2: 00007f09f93313e8 CR3: 00000003fca2a000 CR4: 00000000003506f0
[33119.137325] Call Trace:
[33119.137329]  <TASK>
[33119.137335]  ? __warn+0x80/0x120
[33119.137341]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33119.137398]  ? report_bug+0x164/0x190
[33119.137403]  ? handle_bug+0x3d/0x80
[33119.137408]  ? exc_invalid_op+0x17/0x70
[33119.137412]  ? asm_exc_invalid_op+0x1a/0x20
[33119.137419]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33119.137469]  ? bch2_trans_put+0x23d/0x270 [bcachefs]
[33119.137518]  bch2_readahead+0x314/0x420 [bcachefs]
[33119.137582]  read_pages+0x6a/0x240
[33119.137588]  page_cache_ra_order+0x1f1/0x2e0
[33119.137594]  filemap_get_pages+0xfd/0x620
[33119.137599]  ? _copy_to_iter+0x8b/0x570
[33119.137607]  filemap_read+0xf2/0x360
[33119.137620]  bch2_read_iter+0x1a4/0x640 [bcachefs]
[33119.137681]  ? __pfx_futex_wake_mark+0x10/0x10
[33119.137687]  ? srso_return_thunk+0x5/0x5f
[33119.137695]  vfs_read+0x23a/0x370
[33119.137703]  ksys_read+0x6f/0xf0
[33119.137708]  do_syscall_64+0xb8/0x200
[33119.137712]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[33119.137715] RIP: 0033:0x7ffff7e9c37c
[33119.137749] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 71 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f 72 f8 ff 48
[33119.137751] RSP: 002b:00007fffb35ffc90 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[33119.137754] RAX: ffffffffffffffda RBX: 00007fffb35ffd6c RCX: 00007ffff7e9c37c
[33119.137756] RDX: 0000000000020000 RSI: 00007ffed0079580 RDI: 0000000000000018
[33119.137758] RBP: 00007ffed012f6c8 R08: 0000000000000000 R09: 0000000000000000
[33119.137760] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffed0079580
[33119.137762] R13: 0000000000020000 R14: 00007fffb35ffd0c R15: 00007fffb35ffd6c
[33119.137768]  </TASK>
[33119.137770] ---[ end trace 0000000000000000 ]---
[33139.770571] ------------[ cut here ]------------
[33139.770576] btree trans held srcu lock (delaying memory reclaim) for 49 seconds
[33139.770626] WARNING: CPU: 14 PID: 8782 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
[33139.770706] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class
[33139.770826]  nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic
[33139.770961]  dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4
[33139.770980] CPU: 14 PID: 8782 Comm: nix-store Tainted: G        W  O       6.9.8 #1-NixOS
[33139.770984] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[33139.770987] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
[33139.771058] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
[33139.771061] RSP: 0018:ffffb2a40c447988 EFLAGS: 00010282
[33139.771065] RAX: 0000000000000000 RBX: ffff931dcd3d5200 RCX: 0000000000000027
[33139.771069] RDX: ffff9320cef21848 RSI: 0000000000000001 RDI: ffff9320cef21840
[33139.771078] RBP: ffff931eec1dc000 R08: 0000000000000000 R09: 0000000000000003
[33139.771081] R10: ffffb2a40c447830 R11: ffffffff98f3a128 R12: ffff931eec1dc000
[33139.771083] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000
[33139.771086] FS:  00007ffb5141d500(0000) GS:ffff9320cef00000(0000) knlGS:0000000000000000
[33139.771090] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33139.771093] CR2: 000000c0001d1010 CR3: 00000003f70c0000 CR4: 00000000003506f0
[33139.771096] Call Trace:
[33139.771099]  <TASK>
[33139.771104]  ? __warn+0x80/0x120
[33139.771111]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33139.771183]  ? report_bug+0x164/0x190
[33139.771190]  ? handle_bug+0x3d/0x80
[33139.771196]  ? exc_invalid_op+0x17/0x70
[33139.771201]  ? asm_exc_invalid_op+0x1a/0x20
[33139.771210]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33139.771280]  ? bch2_trans_put+0x23d/0x270 [bcachefs]
[33139.771351]  bch2_readahead+0x314/0x420 [bcachefs]
[33139.771442]  read_pages+0x6a/0x240
[33139.771449]  page_cache_ra_order+0x1f1/0x2e0
[33139.771457]  filemap_get_pages+0xfd/0x620
[33139.771462]  ? _copy_to_iter+0x8b/0x570
[33139.771472]  filemap_read+0xf2/0x360
[33139.771489]  bch2_read_iter+0x1a4/0x640 [bcachefs]
[33139.771577]  ? srso_return_thunk+0x5/0x5f
[33139.771583]  ? srso_return_thunk+0x5/0x5f
[33139.771587]  ? task_tick_fair+0x40/0x3f0
[33139.771592]  ? srso_return_thunk+0x5/0x5f
[33139.771597]  ? srso_return_thunk+0x5/0x5f
[33139.771605]  vfs_read+0x23a/0x370
[33139.771615]  ksys_read+0x6f/0xf0
[33139.771621]  do_syscall_64+0xb8/0x200
[33139.771627]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[33139.771631] RIP: 0033:0x7ffb52d24c1c
[33139.771667] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 19 d9 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 6f d9 f8 ff 48
[33139.771670] RSP: 002b:00007fff7df92370 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[33139.771675] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007ffb52d24c1c
[33139.771678] RDX: 0000000000010000 RSI: 000055daa600b460 RDI: 000000000000000b
[33139.771680] RBP: 000055daa600b460 R08: 0000000000000000 R09: 00000000078bffff
[33139.771683] R10: 000000007ed8320b R11: 0000000000000246 R12: 000000000000000b
[33139.771685] R13: 00007ffb53305768 R14: 00007ffb5141d280 R15: 0000000000000000
[33139.771693]  </TASK>
[33139.771696] ---[ end trace 0000000000000000 ]---
[33141.614991] ------------[ cut here ]------------
[33141.614996] btree trans held srcu lock (delaying memory reclaim) for 51 seconds
[33141.615043] WARNING: CPU: 9 PID: 9250 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
[33141.615112] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class
[33141.615221]  nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic
[33141.615351]  dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4
[33141.615366] CPU: 9 PID: 9250 Comm: ForkJoinPool-1- Tainted: G        W  O       6.9.8 #1-NixOS
[33141.615370] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[33141.615372] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
[33141.615423] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
[33141.615426] RSP: 0018:ffffb2a40eea7990 EFLAGS: 00010282
[33141.615429] RAX: 0000000000000000 RBX: ffff931e27d22400 RCX: 0000000000000027
[33141.615431] RDX: ffff9320ceca1848 RSI: 0000000000000001 RDI: ffff9320ceca1840
[33141.615433] RBP: ffff931ee3264000 R08: 0000000000000000 R09: 0000000000000003
[33141.615435] R10: ffffb2a40eea7838 R11: ffffffff98f3a128 R12: ffff931ee3264000
[33141.615437] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000
[33141.615439] FS:  00007fffbd6006c0(0000) GS:ffff9320cec80000(0000) knlGS:0000000000000000
[33141.615442] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33141.615444] CR2: 000000c0000b5010 CR3: 00000003fca2a000 CR4: 00000000003506f0
[33141.615446] Call Trace:
[33141.615450]  <TASK>
[33141.615454]  ? __warn+0x80/0x120
[33141.615459]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33141.615511]  ? report_bug+0x164/0x190
[33141.615517]  ? handle_bug+0x3d/0x80
[33141.615522]  ? exc_invalid_op+0x17/0x70
[33141.615526]  ? asm_exc_invalid_op+0x1a/0x20
[33141.615533]  ? bch2_trans_put+0x23e/0x270 [bcachefs]
[33141.615584]  bch2_readahead+0x314/0x420 [bcachefs]
[33141.615647]  read_pages+0x6a/0x240
[33141.615654]  page_cache_ra_unbounded+0x140/0x190
[33141.615660]  filemap_get_pages+0x4c3/0x620
[33141.615664]  ? _copy_to_iter+0x8b/0x570
[33141.615668]  ? sugov_update_single_freq+0x14d/0x210
[33141.615677]  filemap_read+0xf2/0x360
[33141.615681]  ? srso_return_thunk+0x5/0x5f
[33141.615694]  bch2_read_iter+0x1a4/0x640 [bcachefs]
[33141.615755]  ? __pfx_futex_wake_mark+0x10/0x10
[33141.615761]  ? srso_return_thunk+0x5/0x5f
[33141.615768]  vfs_read+0x23a/0x370
[33141.615776]  ksys_read+0x6f/0xf0
[33141.615780]  do_syscall_64+0xb8/0x200
[33141.615785]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[33141.615788] RIP: 0033:0x7ffff7e9c37c
[33141.615822] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 71 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f 72 f8 ff 48
[33141.615824] RSP: 002b:00007fffbd5ffc90 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[33141.615827] RAX: ffffffffffffffda RBX: 00007fffbd5ffd6c RCX: 00007ffff7e9c37c
[33141.615829] RDX: 0000000000020000 RSI: 00007ffec4062470 RDI: 000000000000001d
[33141.615831] RBP: 00007ffec40194f8 R08: 0000000000000000 R09: 0000000000000000
[33141.615833] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffec4062470
[33141.615835] R13: 0000000000020000 R14: 00007fffbd5ffd0c R15: 00007fffbd5ffd6c
[33141.615841]  </TASK>
[33141.615843] ---[ end trace 0000000000000000 ]---
koverstreet commented 4 months ago

Those are warnings, not panics - they should be better on 6.10.

I think this bug may be fixed on 6.10, but I need confirmation

On Fri, Jul 12, 2024, 5:37 AM wucke13 @.***> wrote:

So I rebooted the machine, but it got in a kernel panick. I used the HW reset switch to reboot, and ran fsck.bcachefs -p -y -f -k -v /dev/nvme0n1p2. This seems to have run quite uneventful, log below. The machine is now up an running again. I got a couple of other bcachefs related warnings, but IO peformance seems good (much bettern than yesterday), and this only a handful of warnings vs the 50 or so kernel panicks per second yesterday.

I will try to go for 6.10-rc and report back, if I do so.

bcachefs (nvme0n1p2): mounting version 1.7: mi_btree_bitmap opts=ro,degraded,verbose,fsck,fix_errors=yes,read_only bcachefs (nvme0n1p2): recovering from unclean shutdown bcachefs (nvme0n1p2): starting journal read bcachefs (nvme0n1p2): journal read done on device nvme0n1p2, ret 0 bcachefs (nvme0n1p2): journal read done, replaying entries 50236341-50236341 bcachefs (nvme0n1p2): Journal keys: 0 read, 0 after sorting and compacting bcachefs (nvme0n1p2): alloc_read... done bcachefs (nvme0n1p2): stripes_read... done bcachefs (nvme0n1p2): snapshots_read... done bcachefs (nvme0n1p2): check_allocations... done bcachefs (nvme0n1p2): going read-write bcachefs (nvme0n1p2): journal_replay... done bcachefs (nvme0n1p2): check_alloc_info... done bcachefs (nvme0n1p2): check_lrus... done bcachefs (nvme0n1p2): check_btree_backpointers... done bcachefs (nvme0n1p2): check_backpointers_to_extents... done bcachefs (nvme0n1p2): check_extents_to_backpointers... done bcachefs (nvme0n1p2): check_alloc_to_lru_refs... done bcachefs (nvme0n1p2): check_snapshot_trees... done bcachefs (nvme0n1p2): check_snapshots... done bcachefs (nvme0n1p2): check_subvols... done bcachefs (nvme0n1p2): check_subvol_children... done bcachefs (nvme0n1p2): delete_dead_snapshots... done bcachefs (nvme0n1p2): check_inodes... done bcachefs (nvme0n1p2): check_extents... done bcachefs (nvme0n1p2): check_indirect_extents... done bcachefs (nvme0n1p2): check_dirents... done bcachefs (nvme0n1p2): check_xattrs... done bcachefs (nvme0n1p2): check_root... done bcachefs (nvme0n1p2): check_subvolume_structure... done bcachefs (nvme0n1p2): check_directory_structure... done bcachefs (nvme0n1p2): check_nlinks... done bcachefs (nvme0n1p2): resume_logged_ops... done bcachefs (nvme0n1p2): delete_dead_inodes... done bcachefs (nvme0n1p2): going read-only bcachefs (nvme0n1p2): finished waiting for writes to stop bcachefs (nvme0n1p2): flushing journal and stopping allocators, journal seq 50236704 bcachefs (nvme0n1p2): flushing journal and stopping allocators complete, journal seq 50236713 bcachefs (nvme0n1p2): shutdown complete, journal seq 50236714 bcachefs (nvme0n1p2): marking filesystem clean bcachefs (nvme0n1p2): done starting filesystem bcachefs (nvme0n1p2): shutting down bcachefs (nvme0n1p2): shutdown complete

[33102.639053] ------------[ cut here ]------------ [33102.639057] btree trans held srcu lock (delaying memory reclaim) for 10 seconds [33102.639108] WARNING: CPU: 7 PID: 1956 at fs/bcachefs/btree_iter.c:2871 bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs] [33102.639169] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class [33102.639263] nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic [33102.639370] dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4 [33102.639385] CPU: 7 PID: 1956 Comm: zerotier-one Tainted: G O 6.9.8 #1-NixOS [33102.639389] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020 [33102.639391] RIP: 0010:bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs] [33102.639442] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 c6 fc 87 d6 90 <0f> 0b 90 90 e9 5f ff ff ff 90 0f 0b 90 e9 6c ff ff ff 0f 1f 00 90 [33102.639445] RSP: 0018:ffffb2a40163f9b8 EFLAGS: 00010282 [33102.639449] RAX: 0000000000000000 RBX: ffff931e07f34000 RCX: 0000000000000027 [33102.639451] RDX: ffff9320ceba1848 RSI: 0000000000000001 RDI: ffff9320ceba1840 [33102.639453] RBP: ffff931dd50c0000 R08: 0000000000000000 R09: 0000000000000003 [33102.639456] R10: ffffb2a40163f860 R11: ffffffff98f3a128 R12: ffffb2a40163fa80 [33102.639458] R13: ffff931e07f34000 R14: ffff931e07f34000 R15: ffff9320cc55e508 [33102.639460] FS: 00007fc0892006c0(0000) GS:ffff9320ceb80000(0000) knlGS:0000000000000000 [33102.639463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33102.639466] CR2: 00007f5f74bbcce0 CR3: 00000001bb260000 CR4: 00000000003506f0 [33102.639469] Call Trace: [33102.639472] [33102.639477] ? warn+0x80/0x120 [33102.639482] ? bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs] [33102.639534] ? report_bug+0x164/0x190 [33102.639541] ? handle_bug+0x3d/0x80 [33102.639547] ? exc_invalid_op+0x17/0x70 [33102.639551] ? asm_exc_invalid_op+0x1a/0x20 [33102.639558] ? bch2_trans_srcu_unlock+0x11b/0x130 [bcachefs] [33102.639610] ? bch2_trans_srcu_unlock+0x11a/0x130 [bcachefs] [33102.639659] ? bch2_trans_begin+0xf8/0x600 [bcachefs] [33102.639707] bch2_trans_begin+0x5a5/0x600 [bcachefs] [33102.639759] bch2_logged_op_start+0x1d/0xa0 [bcachefs] [33102.639838] bch2_truncate+0xa6/0x110 [bcachefs] [33102.639904] bchfs_truncate+0x274/0x3c0 [bcachefs] [33102.639969] notify_change+0x1f5/0x4c0 [33102.639977] ? do_truncate+0x98/0xf0 [33102.639982] ? srso_return_thunk+0x5/0x5f [33102.639986] do_truncate+0x98/0xf0 [33102.639993] path_openat+0xf96/0x1150 [33102.640001] do_filp_open+0xc4/0x170 [33102.640011] do_sys_openat2+0xab/0xe0 [33102.640014] ? srso_return_thunk+0x5/0x5f [33102.640019] x64_sys_openat+0x57/0xa0 [33102.640023] do_syscall_64+0xb8/0x200 [33102.640028] entry_SYSCALL_64_after_hwframe+0x77/0x7f [33102.640031] RIP: 0033:0x7fc0894a1330 [33102.640066] Code: 48 89 44 24 20 75 94 44 89 54 24 0c e8 09 e2 f8 ff 44 8b 54 24 0c 89 da 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 89 44 24 0c e8 5c e2 f8 ff 8b 44 [33102.640069] RSP: 002b:00007fc0891ff820 EFLAGS: 00000293 ORIG_RAX: 0000000000000101 [33102.640072] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007fc0894a1330 [33102.640074] RDX: 0000000000000241 RSI: 0000561011357590 RDI: 00000000ffffff9c [33102.640076] RBP: 0000561011357590 R08: 0000000000000000 R09: 0000000000000001 [33102.640078] R10: 00000000000001b6 R11: 0000000000000293 R12: 00007fc089864902 [33102.640080] R13: 00007fc089864901 R14: 0000000000000001 R15: 00007fc0891ffa10 [33102.640086] [33102.640088] ---[ end trace 0000000000000000 ]--- [33119.136907] ------------[ cut here ]------------ [33119.136915] btree trans held srcu lock (delaying memory reclaim) for 29 seconds [33119.136960] WARNING: CPU: 13 PID: 9254 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs] [33119.137019] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class [33119.137110] nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic [33119.137215] dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4 [33119.137230] CPU: 13 PID: 9254 Comm: ForkJoinPool-1- Tainted: G W O 6.9.8 #1-NixOS [33119.137234] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020 [33119.137236] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs] [33119.137296] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a [33119.137299] RSP: 0018:ffffb2a400d4b988 EFLAGS: 00010282 [33119.137302] RAX: 0000000000000000 RBX: ffff931f99213a00 RCX: 0000000000000027 [33119.137305] RDX: ffff9320ceea1848 RSI: 0000000000000001 RDI: ffff9320ceea1840 [33119.137307] RBP: ffff931e5c340000 R08: 0000000000000000 R09: 0000000000000003 [33119.137310] R10: ffffb2a400d4b830 R11: ffffffff98f3a128 R12: ffff931e5c340000 [33119.137312] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000 [33119.137314] FS: 00007fffb36006c0(0000) GS:ffff9320cee80000(0000) knlGS:0000000000000000 [33119.137320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33119.137322] CR2: 00007f09f93313e8 CR3: 00000003fca2a000 CR4: 00000000003506f0 [33119.137325] Call Trace: [33119.137329] [33119.137335] ? warn+0x80/0x120 [33119.137341] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33119.137398] ? report_bug+0x164/0x190 [33119.137403] ? handle_bug+0x3d/0x80 [33119.137408] ? exc_invalid_op+0x17/0x70 [33119.137412] ? asm_exc_invalid_op+0x1a/0x20 [33119.137419] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33119.137469] ? bch2_trans_put+0x23d/0x270 [bcachefs] [33119.137518] bch2_readahead+0x314/0x420 [bcachefs] [33119.137582] read_pages+0x6a/0x240 [33119.137588] page_cache_ra_order+0x1f1/0x2e0 [33119.137594] filemap_get_pages+0xfd/0x620 [33119.137599] ? _copy_to_iter+0x8b/0x570 [33119.137607] filemap_read+0xf2/0x360 [33119.137620] bch2_read_iter+0x1a4/0x640 [bcachefs] [33119.137681] ? __pfx_futex_wake_mark+0x10/0x10 [33119.137687] ? srso_return_thunk+0x5/0x5f [33119.137695] vfs_read+0x23a/0x370 [33119.137703] ksys_read+0x6f/0xf0 [33119.137708] do_syscall_64+0xb8/0x200 [33119.137712] entry_SYSCALL_64_after_hwframe+0x77/0x7f [33119.137715] RIP: 0033:0x7ffff7e9c37c [33119.137749] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 71 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f 72 f8 ff 48 [33119.137751] RSP: 002b:00007fffb35ffc90 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [33119.137754] RAX: ffffffffffffffda RBX: 00007fffb35ffd6c RCX: 00007ffff7e9c37c [33119.137756] RDX: 0000000000020000 RSI: 00007ffed0079580 RDI: 0000000000000018 [33119.137758] RBP: 00007ffed012f6c8 R08: 0000000000000000 R09: 0000000000000000 [33119.137760] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffed0079580 [33119.137762] R13: 0000000000020000 R14: 00007fffb35ffd0c R15: 00007fffb35ffd6c [33119.137768] [33119.137770] ---[ end trace 0000000000000000 ]--- [33139.770571] ------------[ cut here ]------------ [33139.770576] btree trans held srcu lock (delaying memory reclaim) for 49 seconds [33139.770626] WARNING: CPU: 14 PID: 8782 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs] [33139.770706] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class [33139.770826] nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic [33139.770961] dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4 [33139.770980] CPU: 14 PID: 8782 Comm: nix-store Tainted: G W O 6.9.8 #1-NixOS [33139.770984] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020 [33139.770987] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs] [33139.771058] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a [33139.771061] RSP: 0018:ffffb2a40c447988 EFLAGS: 00010282 [33139.771065] RAX: 0000000000000000 RBX: ffff931dcd3d5200 RCX: 0000000000000027 [33139.771069] RDX: ffff9320cef21848 RSI: 0000000000000001 RDI: ffff9320cef21840 [33139.771078] RBP: ffff931eec1dc000 R08: 0000000000000000 R09: 0000000000000003 [33139.771081] R10: ffffb2a40c447830 R11: ffffffff98f3a128 R12: ffff931eec1dc000 [33139.771083] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000 [33139.771086] FS: 00007ffb5141d500(0000) GS:ffff9320cef00000(0000) knlGS:0000000000000000 [33139.771090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33139.771093] CR2: 000000c0001d1010 CR3: 00000003f70c0000 CR4: 00000000003506f0 [33139.771096] Call Trace: [33139.771099] [33139.771104] ? warn+0x80/0x120 [33139.771111] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33139.771183] ? report_bug+0x164/0x190 [33139.771190] ? handle_bug+0x3d/0x80 [33139.771196] ? exc_invalid_op+0x17/0x70 [33139.771201] ? asm_exc_invalid_op+0x1a/0x20 [33139.771210] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33139.771280] ? bch2_trans_put+0x23d/0x270 [bcachefs] [33139.771351] bch2_readahead+0x314/0x420 [bcachefs] [33139.771442] read_pages+0x6a/0x240 [33139.771449] page_cache_ra_order+0x1f1/0x2e0 [33139.771457] filemap_get_pages+0xfd/0x620 [33139.771462] ? _copy_to_iter+0x8b/0x570 [33139.771472] filemap_read+0xf2/0x360 [33139.771489] bch2_read_iter+0x1a4/0x640 [bcachefs] [33139.771577] ? srso_return_thunk+0x5/0x5f [33139.771583] ? srso_return_thunk+0x5/0x5f [33139.771587] ? task_tick_fair+0x40/0x3f0 [33139.771592] ? srso_return_thunk+0x5/0x5f [33139.771597] ? srso_return_thunk+0x5/0x5f [33139.771605] vfs_read+0x23a/0x370 [33139.771615] ksys_read+0x6f/0xf0 [33139.771621] do_syscall_64+0xb8/0x200 [33139.771627] entry_SYSCALL_64_after_hwframe+0x77/0x7f [33139.771631] RIP: 0033:0x7ffb52d24c1c [33139.771667] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 19 d9 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 6f d9 f8 ff 48 [33139.771670] RSP: 002b:00007fff7df92370 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [33139.771675] RAX: ffffffffffffffda RBX: 0000000000010000 RCX: 00007ffb52d24c1c [33139.771678] RDX: 0000000000010000 RSI: 000055daa600b460 RDI: 000000000000000b [33139.771680] RBP: 000055daa600b460 R08: 0000000000000000 R09: 00000000078bffff [33139.771683] R10: 000000007ed8320b R11: 0000000000000246 R12: 000000000000000b [33139.771685] R13: 00007ffb53305768 R14: 00007ffb5141d280 R15: 0000000000000000 [33139.771693] [33139.771696] ---[ end trace 0000000000000000 ]--- [33141.614991] ------------[ cut here ]------------ [33141.614996] btree trans held srcu lock (delaying memory reclaim) for 51 seconds [33141.615043] WARNING: CPU: 9 PID: 9250 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs] [33141.615112] Modules linked in: btrfs blake2b_generic vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE ipt_REJECT nf_reject_ipv4 nft_chain_nat nf_nat cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat bridge stp llc amdgpu cfg80211 btusb btrtl btintel btbcm btmtk snd_usb_audio bluetooth snd_usbmidi_lib snd_ump snd_rawmidi snd_seq_device mousedev cp210x joydev input_leds mc ecdh_generic ecc crc16 xt_conntrack snd_hda_codec_realtek nf_conntrack dm_crypt snd_hda_codec_generic nf_defrag_ipv6 nf_defrag_ipv4 snd_hda_scodec_component snd_hda_codec_hdmi edac_mce_amd eeepc_wmi asus_wmi intel_rapl_msr edac_core snd_hda_intel amd_atl ip6t_rpfilter snd_intel_dspcfg battery snd_intel_sdw_acpi platform_profile ipt_rpfilter intel_rapl_common i8042 crc32_pclmul sparse_keymap snd_hda_codec polyval_clmulni polyval_generic amdxcp gf128mul drm_exec ghash_clmulni_intel xt_pkttype snd_hda_core gpu_sched sha512_ssse3 snd_hwdep rfkill drm_buddy xt_LOG evdev sha1_ssse3 mxm_wmi drm_suballoc_helper led_class [33141.615221] nf_log_syslog wmi_bmof cbc mac_hid drm_ttm_helper aesni_intel snd_pcm ttm xt_tcpudp snd_timer crypto_simd encrypted_keys nft_compat drm_display_helper cryptd trusted cec rapl snd sp5100_tco asn1_encoder watchdog tee acpi_cpufreq video k10temp backlight soundcore i2c_piix4 tpm nf_tables tiny_power_button gpio_amdpt sch_fq_codel wmi gpio_generic button uinput hid_xpadneo(O) ff_memless wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel tls atkbd libps2 serio vivaldi_fmap loop sg kvm_amd ccp rng_core kvm fuse configfs efi_pstore nfnetlink zram dmi_sysfs ip_tables x_tables poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod ahci libahci nvme libata igb nvme_core xhci_pci xhci_pci_renesas nvme_auth firmware_class t10_pi ptp crc64_rocksoft xhci_hcd scsi_mod crc32c_intel crc_t10dif pps_core sha256_ssse3 i2c_algo_bit crct10dif_generic [33141.615351] dca crct10dif_pclmul crc64 scsi_common crct10dif_common rtc_cmos tun dm_mod dax af_packet efivarfs autofs4 [33141.615366] CPU: 9 PID: 9250 Comm: ForkJoinPool-1- Tainted: G W O 6.9.8 #1-NixOS [33141.615370] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020 [33141.615372] RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs] [33141.615423] Code: 3d d8 48 c7 c7 d0 c5 b8 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 f3 ef 87 d6 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a [33141.615426] RSP: 0018:ffffb2a40eea7990 EFLAGS: 00010282 [33141.615429] RAX: 0000000000000000 RBX: ffff931e27d22400 RCX: 0000000000000027 [33141.615431] RDX: ffff9320ceca1848 RSI: 0000000000000001 RDI: ffff9320ceca1840 [33141.615433] RBP: ffff931ee3264000 R08: 0000000000000000 R09: 0000000000000003 [33141.615435] R10: ffffb2a40eea7838 R11: ffffffff98f3a128 R12: ffff931ee3264000 [33141.615437] R13: ffff931dd50d5358 R14: ffff931dd50c0000 R15: 0000000000000000 [33141.615439] FS: 00007fffbd6006c0(0000) GS:ffff9320cec80000(0000) knlGS:0000000000000000 [33141.615442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [33141.615444] CR2: 000000c0000b5010 CR3: 00000003fca2a000 CR4: 00000000003506f0 [33141.615446] Call Trace: [33141.615450] [33141.615454] ? __warn+0x80/0x120 [33141.615459] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33141.615511] ? report_bug+0x164/0x190 [33141.615517] ? handle_bug+0x3d/0x80 [33141.615522] ? exc_invalid_op+0x17/0x70 [33141.615526] ? asm_exc_invalid_op+0x1a/0x20 [33141.615533] ? bch2_trans_put+0x23e/0x270 [bcachefs] [33141.615584] bch2_readahead+0x314/0x420 [bcachefs] [33141.615647] read_pages+0x6a/0x240 [33141.615654] page_cache_ra_unbounded+0x140/0x190 [33141.615660] filemap_get_pages+0x4c3/0x620 [33141.615664] ? _copy_to_iter+0x8b/0x570 [33141.615668] ? sugov_update_single_freq+0x14d/0x210 [33141.615677] filemap_read+0xf2/0x360 [33141.615681] ? srso_return_thunk+0x5/0x5f [33141.615694] bch2_read_iter+0x1a4/0x640 [bcachefs] [33141.615755] ? __pfx_futex_wake_mark+0x10/0x10 [33141.615761] ? srso_return_thunk+0x5/0x5f [33141.615768] vfs_read+0x23a/0x370 [33141.615776] ksys_read+0x6f/0xf0 [33141.615780] do_syscall_64+0xb8/0x200 [33141.615785] entry_SYSCALL_64_after_hwframe+0x77/0x7f [33141.615788] RIP: 0033:0x7ffff7e9c37c [33141.615822] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 71 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f 72 f8 ff 48 [33141.615824] RSP: 002b:00007fffbd5ffc90 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [33141.615827] RAX: ffffffffffffffda RBX: 00007fffbd5ffd6c RCX: 00007ffff7e9c37c [33141.615829] RDX: 0000000000020000 RSI: 00007ffec4062470 RDI: 000000000000001d [33141.615831] RBP: 00007ffec40194f8 R08: 0000000000000000 R09: 0000000000000000 [33141.615833] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffec4062470 [33141.615835] R13: 0000000000020000 R14: 00007fffbd5ffd0c R15: 00007fffbd5ffd6c [33141.615841] [33141.615843] ---[ end trace 0000000000000000 ]---

— Reply to this email directly, view it on GitHub https://github.com/koverstreet/bcachefs/issues/693#issuecomment-2225200798, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPGX3SIUQUVH5N6YSNPATLZL6PU5AVCNFSM6AAAAABJRTDW3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGIYDANZZHA . You are receiving this because you were mentioned.Message ID: @.***>

wucke13 commented 3 months ago

@koverstreet I'm on 6.10.0 now, and I could not provoke that issue anymore (heavy read/write work loads where semi-reliable in triggering). However, I did get the following from dmesg: I guess that is totally unrelated to bcachefs.

[  399.600794] SLUB: Unable to allocate memory on node -1, gfp=0xc40(GFP_NOFS)
[  399.600800]   cache: bch_inode_info, object size: 944, buffer size: 944, default order: 3, min order: 0
[  399.600804]   node 0: slabs: 21045, objs: 709140, free: 0
koverstreet commented 3 months ago

that's just memory pressure