search

koverstreet / bcachefs

Other
700 stars 73 forks source link

PC got completely locked up after NFS transfer [b3a782467676e78e78b236049a05ea55af04e83f] #779

Open jpsollie opened 1 week ago

jpsollie commented 1 week ago

To be honest, I have no clue what caused this. I transferred a set of 20x200MB files to my multi-tiered FS. During the transfer, I was already pretty annoyed it took that long, and that the background target had to wake up (which takes some time) before the NFS transfer could be started (whereas the SSD frond/promote targets should perfectly be able to keep it for a while) But, after completely transferring, I saw in dmesg on the target, it wasn't healty:

Nov 11 16:54:29 linuxserver kernel: ------------[ cut here ]------------
Nov 11 16:54:29 linuxserver kernel: btree trans held srcu lock (delaying memory reclaim) for 21 seconds
Nov 11 16:54:29 linuxserver kernel: WARNING: CPU: 17 PID: 2194 at fs/bcachefs/btree_iter.c:3172 bch2_trans_srcu_unlock+0x10d/0x120
Nov 11 16:54:29 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 16:54:29 linuxserver kernel: CPU: 17 UID: 0 PID: 2194 Comm: kworker/u270:0 Not tainted 6.12.0-rc6experimental+ #1
Nov 11 16:54:29 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 16:54:29 linuxserver kernel: Workqueue: bcachefs bch2_write_point_do_index_updates
Nov 11 16:54:29 linuxserver kernel: RIP: 0010:bch2_trans_srcu_unlock+0x10d/0x120
Nov 11 16:54:29 linuxserver kernel: Code: 48 c1 e8 02 48 b9 c3 f5 28 5c 8f c2 f5 28 48 f7 e1 48 c1 ea 02 49 89 fe 48 c7 c7 95 8f 16 82 48 89 d6 e8 f6 ba b5 ff 4c 89 f7 <0f> 0b eb a4 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 55 48 89
Nov 11 16:54:29 linuxserver kernel: RSP: 0018:ffffc900016c3c10 EFLAGS: 00010286
Nov 11 16:54:29 linuxserver kernel: RAX: 0000000000000043 RBX: ffff888129a3c000 RCX: 0000000000000027
Nov 11 16:54:29 linuxserver kernel: RDX: 0000000000000000 RSI: 00000000fffeffff RDI: ffff8882aacc0000
Nov 11 16:54:29 linuxserver kernel: RBP: ffffc900016c3c20 R08: 000000000000ffff R09: ffff88a03f05e240
Nov 11 16:54:29 linuxserver kernel: R10: 000000000002fffd R11: 0000000000000004 R12: ffff8881397ff800
Nov 11 16:54:29 linuxserver kernel: R13: ffff888129a3c000 R14: ffff8882aacc0000 R15: 00000000fffff784
Nov 11 16:54:29 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe440000(0000) knlGS:0000000000000000
Nov 11 16:54:29 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 16:54:29 linuxserver kernel: CR2: 00007ffd08d39a2a CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 16:54:29 linuxserver kernel: Call Trace:
Nov 11 16:54:29 linuxserver kernel:  <TASK>
Nov 11 16:54:29 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 16:54:29 linuxserver kernel:  ? __warn+0xcc/0x170
Nov 11 16:54:29 linuxserver kernel:  ? report_bug+0x144/0x1d0
Nov 11 16:54:29 linuxserver kernel:  ? handle_bug+0x62/0x90
Nov 11 16:54:29 linuxserver kernel:  ? exc_invalid_op+0x1b/0x50
Nov 11 16:54:29 linuxserver kernel:  ? asm_exc_invalid_op+0x1b/0x20
Nov 11 16:54:29 linuxserver kernel:  ? bch2_trans_srcu_unlock+0x10d/0x120
Nov 11 16:54:29 linuxserver kernel:  bch2_trans_begin+0x4a8/0x4c0
Nov 11 16:54:29 linuxserver kernel:  ? bch2_path_put+0x2f9/0x330
Nov 11 16:54:29 linuxserver kernel:  __bch2_write_index+0x111/0x5e0
Nov 11 16:54:29 linuxserver kernel:  bch2_write_point_do_index_updates+0xd8/0x180
Nov 11 16:54:29 linuxserver kernel:  process_scheduled_works+0x1a0/0x2f0
Nov 11 16:54:29 linuxserver kernel:  worker_thread+0x2df/0x370
Nov 11 16:54:29 linuxserver kernel:  kthread+0x112/0x130
Nov 11 16:54:29 linuxserver kernel:  ? pr_cont_work+0x1c0/0x1c0
Nov 11 16:54:29 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 16:54:29 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 16:54:29 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 16:54:29 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 16:54:29 linuxserver kernel:  </TASK>
Nov 11 16:54:29 linuxserver kernel: ---[ end trace 0000000000000000 ]---
Nov 11 16:59:59 linuxserver kernel: elogind-daemon[2657]: Removed session c1.
Nov 11 17:02:33 linuxserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Nov 11 17:02:33 linuxserver kernel: rcu: \x094-....: (2100 ticks this GP) idle=a5ac/1/0x4000000000000000 softirq=6770/6783 fqs=750
Nov 11 17:02:33 linuxserver kernel: rcu: \x09(t=2100 jiffies g=93801 q=6506 ncpus=64)
Nov 11 17:02:33 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W          6.12.0-rc6experimental+ #1
Nov 11 17:02:33 linuxserver kernel: Tainted: [W]=WARN
Nov 11 17:02:33 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:02:33 linuxserver kernel: RIP: 0010:__bch2_inode_hash_find+0xf1/0x150
Nov 11 17:02:33 linuxserver kernel: Code: 0f b7 97 56 67 02 00 49 f7 da eb 16 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 4d 8b 09 41 f6 c1 01 75 c7 4b 39 b4 11 60 02 00 00 <75> ed 4b 39 94 11 68 02 00 00 75 e3 eb 34 48 8b 5b 30 48 85 db 74
Nov 11 17:02:33 linuxserver kernel: RSP: 0018:ffffc900043a77b0 EFLAGS: 00000206
Nov 11 17:02:33 linuxserver kernel: RAX: 0000000000000000 RBX: ffff88844c4f8000 RCX: ffff88844c4f9c38
Nov 11 17:02:33 linuxserver kernel: RDX: 0000000000001000 RSI: 0000000000000013 RDI: ffff8882aacc0000
Nov 11 17:02:33 linuxserver kernel: RBP: ffffc900043a77d8 R08: ffff88844c4f9c39 R09: ffff88811aa41c60
Nov 11 17:02:33 linuxserver kernel: R10: fffffffffffffda8 R11: 000000006af13354 R12: ffff888463994000
Nov 11 17:02:33 linuxserver kernel: R13: ffff88810518e6c0 R14: 0000000000000000 R15: ffff888463994000
Nov 11 17:02:33 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:02:33 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:02:33 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:02:33 linuxserver kernel: Call Trace:
Nov 11 17:02:33 linuxserver kernel:  <IRQ>
Nov 11 17:02:33 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:02:33 linuxserver kernel:  ? dump_cpu_task+0x32/0x60
Nov 11 17:02:33 linuxserver kernel:  ? rcu_dump_cpu_stacks+0xdf/0x130
Nov 11 17:02:33 linuxserver kernel:  ? print_cpu_stall+0x163/0x280
Nov 11 17:02:33 linuxserver kernel:  ? rcu_sched_clock_irq+0x214/0x4b0
Nov 11 17:02:33 linuxserver kernel:  ? update_process_times+0x70/0xa0
Nov 11 17:02:33 linuxserver kernel:  ? tick_nohz_handler+0xbf/0x110
Nov 11 17:02:33 linuxserver kernel:  ? tick_setup_sched_timer+0x180/0x180
Nov 11 17:02:33 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:02:33 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:02:33 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:02:33 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:02:33 linuxserver kernel:  </IRQ>
Nov 11 17:02:33 linuxserver kernel:  <TASK>
Nov 11 17:02:33 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:02:33 linuxserver kernel:  ? __bch2_inode_hash_find+0xf1/0x150
Nov 11 17:02:33 linuxserver kernel:  bch2_inode_hash_find+0x2b/0x1b0
Nov 11 17:02:33 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:02:33 linuxserver kernel:  bch2_inode_hash_insert+0x374/0x450
Nov 11 17:02:33 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:02:33 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:02:33 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:02:33 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:02:33 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:02:33 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:02:33 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:02:33 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:02:33 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:02:33 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:02:33 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:02:33 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:02:33 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:02:33 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:02:33 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:02:33 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:02:33 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:02:33 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:02:33 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:02:33 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:02:33 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:02:33 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:02:33 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:02:33 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:02:33 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:02:33 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:02:33 linuxserver kernel:  </TASK>
Nov 11 17:02:57 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 45s! [nfsd:2971]
Nov 11 17:02:57 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:02:57 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W          6.12.0-rc6experimental+ #1
Nov 11 17:02:57 linuxserver kernel: Tainted: [W]=WARN
Nov 11 17:02:57 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:02:57 linuxserver kernel: RIP: 0010:bch2_vfs_inode_obj_hash_fn.llvm.1337315023117572408+0x93/0xc0
Nov 11 17:02:57 linuxserver kernel: Code: 01 f0 41 01 d0 0f b6 97 68 02 00 00 44 01 c2 31 c8 89 ce c1 c6 0e 29 f0 31 c2 89 c6 c1 c6 0b 29 f2 31 d1 89 d6 c1 c6 19 29 f1 <31> c8 89 ce c1 c6 10 29 f0 31 c2 89 c6 c1 c6 04 29 f2 31 d1 c1 c2
Nov 11 17:02:57 linuxserver kernel: RSP: 0018:ffffc900043a77e8 EFLAGS: 00000287
Nov 11 17:02:57 linuxserver kernel: RAX: 00000000a073809e RBX: ffff888374c38000 RCX: 00000000e2f87d6a
Nov 11 17:02:57 linuxserver kernel: RDX: 00000000c1f519ac RSI: 000000005983ea33 RDI: ffff88810518e468
Nov 11 17:02:57 linuxserver kernel: RBP: ffffc900043a77e8 R08: 00000000fd898e31 R09: 0000000000000000
Nov 11 17:02:57 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:02:57 linuxserver kernel: R13: ffff88844c4f9c38 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:02:57 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:02:57 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:02:57 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:02:57 linuxserver kernel: Call Trace:
Nov 11 17:02:57 linuxserver kernel:  <IRQ>
Nov 11 17:02:57 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:02:57 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:02:57 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:02:57 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:02:57 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:02:57 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:02:57 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:02:57 linuxserver kernel:  </IRQ>
Nov 11 17:02:57 linuxserver kernel:  <TASK>
Nov 11 17:02:57 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:02:57 linuxserver kernel:  ? bch2_vfs_inode_obj_hash_fn.llvm.1337315023117572408+0x93/0xc0
Nov 11 17:02:57 linuxserver kernel:  ? bch2_inode_hash_find+0x2b/0x1b0
Nov 11 17:02:57 linuxserver kernel:  rhashtable_insert_slow+0x95/0x4f0
Nov 11 17:02:57 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:02:57 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:02:57 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:02:57 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:02:57 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:02:57 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:02:57 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:02:57 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:02:57 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:02:57 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:02:57 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:02:57 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:02:57 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:02:57 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:02:57 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:02:57 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:02:57 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:02:57 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:02:57 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:02:57 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:02:57 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:02:57 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:02:57 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:02:57 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:02:57 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:02:57 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:02:57 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:02:57 linuxserver kernel:  </TASK>
Nov 11 17:03:25 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 71s! [nfsd:2971]
Nov 11 17:03:25 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:03:25 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:03:25 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:03:25 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:03:25 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:03:25 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:03:25 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:03:25 linuxserver kernel: RAX: fffffffffffffff5 RBX: ffff888374c38000 RCX: 0000000000000019
Nov 11 17:03:25 linuxserver kernel: RDX: 0000000000000019 RSI: ffff88810518a8e8 RDI: ffffc900043a7810
Nov 11 17:03:25 linuxserver kernel: RBP: ffffc900043a7868 R08: 00000000fd898e31 R09: 0000000000000000
Nov 11 17:03:25 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:03:25 linuxserver kernel: R13: ffff888374c3be40 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:03:25 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:03:25 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:03:25 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:03:25 linuxserver kernel: Call Trace:
Nov 11 17:03:25 linuxserver kernel:  <IRQ>
Nov 11 17:03:25 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:03:25 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:03:25 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:03:25 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:03:25 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:03:25 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:03:25 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:03:25 linuxserver kernel:  </IRQ>
Nov 11 17:03:25 linuxserver kernel:  <TASK>
Nov 11 17:03:25 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:03:25 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:03:25 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:03:25 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:03:25 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:03:25 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:03:25 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:03:25 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:03:25 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:03:25 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:03:25 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:03:25 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:03:25 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:03:25 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:03:25 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:03:25 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:03:25 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:03:25 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:03:25 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:03:25 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:03:25 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:03:25 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:03:25 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:03:25 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:03:25 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:03:25 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:03:25 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:03:25 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:03:25 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:03:25 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:03:25 linuxserver kernel:  </TASK>
Nov 11 17:03:36 linuxserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Nov 11 17:03:36 linuxserver kernel: rcu: \x094-....: (8404 ticks this GP) idle=a5ac/1/0x4000000000000000 softirq=6770/6783 fqs=2985
Nov 11 17:03:36 linuxserver kernel: rcu: \x09(t=8404 jiffies g=93801 q=9443 ncpus=64)
Nov 11 17:03:36 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:03:36 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:03:36 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:03:36 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:03:36 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:03:36 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:03:36 linuxserver kernel: RAX: fffffffffffffff5 RBX: ffff888374c38000 RCX: 0000000000000019
Nov 11 17:03:36 linuxserver kernel: RDX: 0000000000000019 RSI: ffff88810518a8e8 RDI: ffffc900043a7810
Nov 11 17:03:36 linuxserver kernel: RBP: ffffc900043a7868 R08: 00000000fd898e31 R09: 0000000000000000
Nov 11 17:03:36 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:03:36 linuxserver kernel: R13: ffff888374c3be40 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:03:36 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:03:36 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:03:36 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:03:36 linuxserver kernel: Call Trace:
Nov 11 17:03:36 linuxserver kernel:  <IRQ>
Nov 11 17:03:36 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:03:36 linuxserver kernel:  ? dump_cpu_task+0x32/0x60
Nov 11 17:03:36 linuxserver kernel:  ? rcu_dump_cpu_stacks+0xdf/0x130
Nov 11 17:03:36 linuxserver kernel:  ? print_cpu_stall+0x163/0x280
Nov 11 17:03:36 linuxserver kernel:  ? rcu_sched_clock_irq+0x214/0x4b0
Nov 11 17:03:36 linuxserver kernel:  ? update_process_times+0x70/0xa0
Nov 11 17:03:36 linuxserver kernel:  ? tick_nohz_handler+0xbf/0x110
Nov 11 17:03:36 linuxserver kernel:  ? tick_setup_sched_timer+0x180/0x180
Nov 11 17:03:36 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:03:36 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:03:36 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:03:36 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:03:36 linuxserver kernel:  </IRQ>
Nov 11 17:03:36 linuxserver kernel:  <TASK>
Nov 11 17:03:36 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:03:36 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:03:36 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:03:36 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:03:36 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:03:36 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:03:36 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:03:36 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:03:36 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:03:36 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:03:36 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:03:36 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:03:36 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:03:36 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:03:36 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:03:36 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:03:36 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:03:36 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:03:36 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:03:36 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:03:36 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:03:36 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:03:36 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:03:36 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:03:36 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:03:36 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:03:36 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:03:36 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:03:36 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:03:36 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:03:36 linuxserver kernel:  </TASK>
Nov 11 17:04:01 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 105s! [nfsd:2971]
Nov 11 17:04:01 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:04:01 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:04:01 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:04:01 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:04:01 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:01 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:04:01 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:04:01 linuxserver kernel: RAX: ffff888374c38000 RBX: ffff88844c4f8000 RCX: 0000000000000033
Nov 11 17:04:01 linuxserver kernel: RDX: 0000000000000033 RSI: ffff88811aa437c8 RDI: ffffc900043a7810
Nov 11 17:04:01 linuxserver kernel: RBP: ffffc900043a7868 R08: 0000000062a93c84 R09: 0000000000000000
Nov 11 17:04:01 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:04:01 linuxserver kernel: R13: ffff88844c4f9c38 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:04:01 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:04:01 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:04:01 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:04:01 linuxserver kernel: Call Trace:
Nov 11 17:04:01 linuxserver kernel:  <IRQ>
Nov 11 17:04:01 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:04:01 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:04:01 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:04:01 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:04:01 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:04:01 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:04:01 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:04:01 linuxserver kernel:  </IRQ>
Nov 11 17:04:01 linuxserver kernel:  <TASK>
Nov 11 17:04:01 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:04:01 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:01 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:04:01 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:04:01 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:04:01 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:04:01 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:04:01 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:04:01 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:04:01 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:04:01 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:04:01 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:04:01 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:04:01 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:04:01 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:04:01 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:04:01 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:04:01 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:04:01 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:04:01 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:04:01 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:04:01 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:04:01 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:04:01 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:04:01 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:04:01 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:04:01 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:01 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:04:01 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:01 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:04:01 linuxserver kernel:  </TASK>
Nov 11 17:04:29 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 131s! [nfsd:2971]
Nov 11 17:04:29 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:04:29 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:04:29 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:04:29 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:04:29 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:29 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:04:29 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:04:29 linuxserver kernel: RAX: ffff888374c38000 RBX: ffff88844c4f8000 RCX: 0000000000000033
Nov 11 17:04:29 linuxserver kernel: RDX: 0000000000000033 RSI: ffff88811aa437c8 RDI: ffffc900043a7810
Nov 11 17:04:29 linuxserver kernel: RBP: ffffc900043a7868 R08: 0000000062a93c84 R09: 0000000000000000
Nov 11 17:04:29 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:04:29 linuxserver kernel: R13: ffff88844c4f9c38 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:04:29 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:04:29 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:04:29 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:04:29 linuxserver kernel: Call Trace:
Nov 11 17:04:29 linuxserver kernel:  <IRQ>
Nov 11 17:04:29 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:04:29 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:04:29 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:04:29 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:04:29 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:04:29 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:04:29 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:04:29 linuxserver kernel:  </IRQ>
Nov 11 17:04:29 linuxserver kernel:  <TASK>
Nov 11 17:04:29 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:04:29 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:29 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:04:29 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:04:29 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:04:29 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:04:29 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:04:29 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:04:29 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:04:29 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:04:29 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:04:29 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:04:29 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:04:29 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:04:29 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:04:29 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:04:29 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:04:29 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:04:29 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:04:29 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:04:29 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:04:29 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:04:29 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:04:29 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:04:29 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:04:29 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:04:29 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:29 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:04:29 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:29 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:04:29 linuxserver kernel:  </TASK>
Nov 11 17:04:39 linuxserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Nov 11 17:04:39 linuxserver kernel: rcu: \x094-....: (14708 ticks this GP) idle=a5ac/1/0x4000000000000000 softirq=6770/6783 fqs=5221
Nov 11 17:04:39 linuxserver kernel: rcu: \x09(t=14708 jiffies g=93801 q=15788 ncpus=64)
Nov 11 17:04:39 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:04:39 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:04:39 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:04:39 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:39 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:04:39 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:04:39 linuxserver kernel: RAX: fffffffffffffff5 RBX: ffff888374c38000 RCX: 0000000000000019
Nov 11 17:04:39 linuxserver kernel: RDX: 0000000000000019 RSI: ffff88810518a8e8 RDI: ffffc900043a7810
Nov 11 17:04:39 linuxserver kernel: RBP: ffffc900043a7868 R08: 00000000fd898e31 R09: 0000000000000000
Nov 11 17:04:39 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:04:39 linuxserver kernel: R13: ffff888374c3be40 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:04:39 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:04:39 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:04:39 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:04:39 linuxserver kernel: Call Trace:
Nov 11 17:04:39 linuxserver kernel:  <IRQ>
Nov 11 17:04:39 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:04:39 linuxserver kernel:  ? dump_cpu_task+0x32/0x60
Nov 11 17:04:39 linuxserver kernel:  ? rcu_dump_cpu_stacks+0xdf/0x130
Nov 11 17:04:39 linuxserver kernel:  ? print_cpu_stall+0x163/0x280
Nov 11 17:04:39 linuxserver kernel:  ? rcu_sched_clock_irq+0x214/0x4b0
Nov 11 17:04:39 linuxserver kernel:  ? update_process_times+0x70/0xa0
Nov 11 17:04:39 linuxserver kernel:  ? tick_nohz_handler+0xbf/0x110
Nov 11 17:04:39 linuxserver kernel:  ? tick_setup_sched_timer+0x180/0x180
Nov 11 17:04:39 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:04:39 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:04:39 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:04:39 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:04:39 linuxserver kernel:  </IRQ>
Nov 11 17:04:39 linuxserver kernel:  <TASK>
Nov 11 17:04:39 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:04:39 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:04:39 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:04:39 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:04:39 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:04:39 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:04:39 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:04:39 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:04:39 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:04:39 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:04:39 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:04:39 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:04:39 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:04:39 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:04:39 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:04:39 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:04:39 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:04:39 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:04:39 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:04:39 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:04:39 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:04:39 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:04:39 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:04:39 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:04:39 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:04:39 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:04:39 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:39 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:04:39 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:04:39 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:04:39 linuxserver kernel:  </TASK>
Nov 11 17:05:05 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 164s! [nfsd:2971]
Nov 11 17:05:05 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:05:05 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:05:05 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:05:05 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:05:05 linuxserver kernel: RIP: 0010:__bch2_inode_hash_find+0xf1/0x150
Nov 11 17:05:05 linuxserver kernel: Code: 0f b7 97 56 67 02 00 49 f7 da eb 16 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 4d 8b 09 41 f6 c1 01 75 c7 4b 39 b4 11 60 02 00 00 <75> ed 4b 39 94 11 68 02 00 00 75 e3 eb 34 48 8b 5b 30 48 85 db 74
Nov 11 17:05:05 linuxserver kernel: RSP: 0018:ffffc900043a77b0 EFLAGS: 00000206
Nov 11 17:05:05 linuxserver kernel: RAX: 0000000000000000 RBX: ffff88844c4f8000 RCX: ffff88844c4f9c38
Nov 11 17:05:05 linuxserver kernel: RDX: 0000000000001000 RSI: 0000000000000013 RDI: ffff8882aacc0000
Nov 11 17:05:05 linuxserver kernel: RBP: ffffc900043a77d8 R08: ffff88844c4f9c39 R09: ffff88811aa414f0
Nov 11 17:05:05 linuxserver kernel: R10: fffffffffffffda8 R11: 000000006af13354 R12: ffff888463994000
Nov 11 17:05:05 linuxserver kernel: R13: ffff88810518e6c0 R14: 0000000000000000 R15: ffff888463994000
Nov 11 17:05:05 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:05:05 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:05:05 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:05:05 linuxserver kernel: Call Trace:
Nov 11 17:05:05 linuxserver kernel:  <IRQ>
Nov 11 17:05:05 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:05:05 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:05:05 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:05:05 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:05:05 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:05:05 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:05:05 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:05:05 linuxserver kernel:  </IRQ>
Nov 11 17:05:05 linuxserver kernel:  <TASK>
Nov 11 17:05:05 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:05:05 linuxserver kernel:  ? __bch2_inode_hash_find+0xf1/0x150
Nov 11 17:05:05 linuxserver kernel:  bch2_inode_hash_find+0x2b/0x1b0
Nov 11 17:05:05 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:05:05 linuxserver kernel:  bch2_inode_hash_insert+0x374/0x450
Nov 11 17:05:05 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:05:05 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:05:05 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:05:05 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:05:05 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:05:05 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:05:05 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:05:05 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:05:05 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:05:05 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:05:05 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:05:05 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:05:05 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:05:05 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:05:05 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:05:05 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:05:05 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:05:05 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:05:05 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:05:05 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:05:05 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:05:05 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:05:05 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:05:05 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:05:05 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:05:05 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:05:05 linuxserver kernel:  </TASK>
Nov 11 17:05:33 linuxserver kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 190s! [nfsd:2971]
Nov 11 17:05:33 linuxserver kernel: Modules linked in: amdgpu drm_buddy drm_suballoc_helper drm_exec binfmt_misc drm_display_helper gpu_sched polyval_clmulni amdxcp polyval_generic nct6775 drm_ttm_helper hwmon_vid sha512_ssse3 sr_mod sha256_ssse3 ttm nct6775_core cdrom atlantic hid_generic sha1_ssse3 mpt3sas wmi_bmof ccp i2c_piix4 i2c_smbus tpm_crb tpm_tis tpm_tis_core efivarfs
Nov 11 17:05:33 linuxserver kernel: CPU: 4 UID: 0 PID: 2971 Comm: nfsd Tainted: G        W    L     6.12.0-rc6experimental+ #1
Nov 11 17:05:33 linuxserver kernel: Tainted: [W]=WARN, [L]=SOFTLOCKUP
Nov 11 17:05:33 linuxserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
Nov 11 17:05:33 linuxserver kernel: RIP: 0010:rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:05:33 linuxserver kernel: Code: f8 9f ff 31 c0 eb 10 48 8b 43 30 48 85 c0 49 0f 44 c4 eb 03 4c 89 e0 f0 41 80 65 00 fe f7 45 b8 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 8d 00 00 00 48 3d 00 f0 ff ff 0f 86 21 fd ff ff eb 7f 8b 53
Nov 11 17:05:33 linuxserver kernel: RSP: 0018:ffffc900043a77f8 EFLAGS: 00000286
Nov 11 17:05:33 linuxserver kernel: RAX: ffff888374c38000 RBX: ffff88844c4f8000 RCX: 0000000000000033
Nov 11 17:05:33 linuxserver kernel: RDX: 0000000000000033 RSI: ffff88811aa437c8 RDI: ffffc900043a7810
Nov 11 17:05:33 linuxserver kernel: RBP: ffffc900043a7868 R08: 0000000062a93c84 R09: 0000000000000000
Nov 11 17:05:33 linuxserver kernel: R10: 0000000062a93c84 R11: ffff8882aace6740 R12: fffffffffffffff5
Nov 11 17:05:33 linuxserver kernel: R13: ffff88844c4f9c38 R14: ffff88810518e6c0 R15: ffff8882aace6740
Nov 11 17:05:33 linuxserver kernel: FS:  0000000000000000(0000) GS:ffff889fbe100000(0000) knlGS:0000000000000000
Nov 11 17:05:33 linuxserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 17:05:33 linuxserver kernel: CR2: 0000564f27074af0 CR3: 0000000002609000 CR4: 00000000003506f0
Nov 11 17:05:33 linuxserver kernel: Call Trace:
Nov 11 17:05:33 linuxserver kernel:  <IRQ>
Nov 11 17:05:33 linuxserver kernel:  ? show_regs+0x59/0x60
Nov 11 17:05:33 linuxserver kernel:  ? watchdog_timer_fn+0x1d9/0x250
Nov 11 17:05:33 linuxserver kernel:  ? lockup_detector_soft_poweroff+0x20/0x20
Nov 11 17:05:33 linuxserver kernel:  ? __hrtimer_run_queues+0xd5/0x160
Nov 11 17:05:33 linuxserver kernel:  ? hrtimer_interrupt+0xed/0x380
Nov 11 17:05:33 linuxserver kernel:  ? __sysvec_apic_timer_interrupt+0x41/0xb0
Nov 11 17:05:33 linuxserver kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x80
Nov 11 17:05:33 linuxserver kernel:  </IRQ>
Nov 11 17:05:33 linuxserver kernel:  <TASK>
Nov 11 17:05:33 linuxserver kernel:  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 17:05:33 linuxserver kernel:  ? rhashtable_insert_slow+0x33d/0x4f0
Nov 11 17:05:33 linuxserver kernel:  ? rhashtable_insert_slow+0x1f3/0x4f0
Nov 11 17:05:33 linuxserver kernel:  bch2_inode_hash_insert+0x2e8/0x450
Nov 11 17:05:33 linuxserver kernel:  bch2_inode_hash_init_insert+0x65/0xe0
Nov 11 17:05:33 linuxserver kernel:  bch2_lookup+0x567/0x730
Nov 11 17:05:33 linuxserver kernel:  ? bch2_subvolume_get+0x143/0x190
Nov 11 17:05:33 linuxserver kernel:  ? kmem_cache_alloc_lru_noprof+0xfa/0x180
Nov 11 17:05:33 linuxserver kernel:  ? __d_alloc.llvm.7640851717847284446+0x155/0x1b0
Nov 11 17:05:33 linuxserver kernel:  __lookup_slow+0xc4/0x120
Nov 11 17:05:33 linuxserver kernel:  lookup_one_unlocked+0x8b/0xb0
Nov 11 17:05:33 linuxserver kernel:  lookup_positive_unlocked+0x18/0x40
Nov 11 17:05:33 linuxserver kernel:  nfs3svc_encode_entryplus3+0x115/0x290
Nov 11 17:05:33 linuxserver kernel:  nfsd_readdir+0x10b/0x2f0
Nov 11 17:05:33 linuxserver kernel:  ? svcxdr_encode_entry3_common+0xd0/0xd0
Nov 11 17:05:33 linuxserver kernel:  ? nfsd_splice_actor+0xc0/0xc0
Nov 11 17:05:33 linuxserver kernel:  nfsd3_proc_readdirplus+0x1be/0x250
Nov 11 17:05:33 linuxserver kernel:  nfsd_dispatch+0xa3/0x130
Nov 11 17:05:33 linuxserver kernel:  svc_process_common+0x309/0x4d0
Nov 11 17:05:33 linuxserver kernel:  ? nfsd_svc+0x290/0x290
Nov 11 17:05:33 linuxserver kernel:  svc_process+0x107/0x150
Nov 11 17:05:33 linuxserver kernel:  svc_recv+0x9a6/0xa20
Nov 11 17:05:33 linuxserver kernel:  ? nfsd_file_net_dispose+0x1de/0x200
Nov 11 17:05:33 linuxserver kernel:  nfsd+0xc1/0xf0
Nov 11 17:05:33 linuxserver kernel:  kthread+0x112/0x130
Nov 11 17:05:33 linuxserver kernel:  ? nfsd_current_rqst+0x30/0x30
Nov 11 17:05:33 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:05:33 linuxserver kernel:  ret_from_fork+0x31/0x40
Nov 11 17:05:33 linuxserver kernel:  ? kthread_blkcg+0x30/0x30
Nov 11 17:05:33 linuxserver kernel:  ret_from_fork_asm+0x11/0x20
Nov 11 17:05:33 linuxserver kernel:  </TASK>

after a while, I was unable to login again. No SSH, no samba mounts, not native terminal, nothing. I also wasn't able to unmount the NFS share on the target anymore. The magic sysrq+B keyboard combination made my PC reboot, but for the rest nothing was possible anymore.

jpsollie commented 1 week ago

Decoder stacktrace is here: error.txt

jpsollie commented 22 hours ago

apparently, it's not only NFS: a simple "ls .snapshots" (.snapshots is the directory where my snapshots live) is enough to trigger the bug:

[  267.673698] [   C22] rcu: INFO: rcu_sched self-detected stall on CPU
[  267.673701] [   C22] rcu:    22-....: (2099 ticks this GP) idle=fb04/1/0x4000000000000000 softirq=1207/1207 fqs=1043
[  267.673706] [   C22] rcu:    (t=2100 jiffies g=2049 q=339 ncpus=64)
[  267.673708] [   C22] CPU: 22 UID: 0 PID: 4929 Comm: ls Not tainted 6.12.0release+ #4
[  267.673709] [   C22] Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
[  267.673710] [   C22] RIP: 0010:rhashtable_insert_slow+0x349/0x500
[  267.673714] [   C22] Code: 95 ff 31 c0 eb 10 49 8b 45 30 48 85 c0 49 0f 44 c6 eb 03 4c 89 f0 f0 41 80 24 24 fe f7 44 24 38 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 91 00 00 00 48 3d 00 f0 ff ff 0f 86 15 fd ff ff e9 80 00 00
[  267.673716] [   C22] RSP: 0018:ffff8882155b3870 EFLAGS: 00000286
[  267.673717] [   C22] RAX: ffff88810462d800 RBX: ffff88810e571551 RCX: 0000000000000011
[  267.673718] [   C22] RDX: 0000000000000011 RSI: ffff888259978000 RDI: ffff8882155b3888
[  267.673719] [   C22] RBP: ffff88823cbc7958 R08: 000000008e0492de R09: 0000000000000000
[  267.673720] [   C22] R10: 000000008e0492de R11: ffff88821c326680 R12: ffff88810e571550
[  267.673721] [   C22] R13: ffff88810e571400 R14: fffffffffffffff5 R15: ffff88821c326680
[  267.673721] [   C22] FS:  00007fb9de145740(0000) GS:ffff889fbe580000(0000) knlGS:0000000000000000
[  267.673722] [   C22] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  267.673723] [   C22] CR2: 000055da12573040 CR3: 000000021542d000 CR4: 00000000003506f0
[  267.673724] [   C22] Call Trace:
[  267.673725] [   C22]  <IRQ>
[  267.673726] [   C22]  ? rcu_dump_cpu_stacks+0xdf/0x130
[  267.673729] [   C22]  ? print_cpu_stall+0x173/0x2c0
[  267.673730] [   C22]  ? rcu_sched_clock_irq+0x221/0x5d0
[  267.673731] [   C22]  ? update_process_times+0x71/0xa0
[  267.673733] [   C22]  ? tick_nohz_handler+0xbb/0x110
[  267.673735] [   C22]  ? tick_setup_sched_timer+0x180/0x180
[  267.673736] [   C22]  ? __hrtimer_run_queues+0xf1/0x250
[  267.673737] [   C22]  ? hrtimer_interrupt+0xf0/0x390
[  267.673738] [   C22]  ? __sysvec_apic_timer_interrupt+0x47/0x140
[  267.673740] [   C22]  ? sysvec_apic_timer_interrupt+0x63/0x80
[  267.673741] [   C22]  </IRQ>
[  267.673742] [   C22]  <TASK>
[  267.673742] [   C22]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[  267.673744] [   C22]  ? rhashtable_insert_slow+0x349/0x500
[  267.673745] [   C22]  bch2_inode_hash_insert+0x309/0x480
[  267.673747] [   C22]  bch2_lookup+0x567/0x730
[  267.673749] [   C22]  ? __memcg_slab_post_alloc_hook+0x2ca/0x370
[  267.673751] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x2d/0x1b0
[  267.673753] [   C22]  ? kmem_cache_alloc_lru_noprof+0xe9/0x1b0
[  267.673754] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x14e/0x1b0
[  267.673756] [   C22]  __lookup_slow+0xcb/0x130
[  267.673758] [   C22]  lookup_slow+0x33/0x50
[  267.673759] [   C22]  walk_component+0xcc/0xe0
[  267.673761] [   C22]  path_lookupat+0x4d/0xf0
[  267.673762] [   C22]  filename_lookup+0xa8/0x140
[  267.673764] [   C22]  vfs_statx+0x64/0xf0
[  267.673765] [   C22]  __se_sys_statx+0xc9/0x180
[  267.673766] [   C22]  ? touch_atime+0x23/0x1b0
[  267.673767] [   C22]  ? bch2_vfs_readdir+0xc2/0x160
[  267.673769] [   C22]  ? iterate_dir+0x92/0x120
[  267.673770] [   C22]  ? __se_sys_getdents64+0xa4/0xd0
[  267.673771] [   C22]  ? filldir+0x180/0x180
[  267.673772] [   C22]  do_syscall_64+0x52/0xf0
[  267.673774] [   C22]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  267.673775] [   C22] RIP: 0033:0x7fb9de26884e
[  267.673777] [   C22] Code: 05 0c 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 9a fd ff ff e8 d3 88 01 00 0f 1f 00 f3 0f 1e fa 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2a 89 c1 85 c0 74 0f 48 8b 05 a5 05 0c 00 64
[  267.673778] [   C22] RSP: 002b:00007ffff17b71f8 EFLAGS: 00000246 ORIG_RAX: 000000000000014c
[  267.673779] [   C22] RAX: ffffffffffffffda RBX: 000055da12573be8 RCX: 00007fb9de26884e
[  267.673780] [   C22] RDX: 0000000000000900 RSI: 00007ffff17b7340 RDI: 00000000ffffff9c
[  267.673780] [   C22] RBP: 00007ffff17b7330 R08: 00007ffff17b7210 R09: 00007ffff17b73e0
[  267.673781] [   C22] R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000002
[  267.673781] [   C22] R13: 000055da12573bd0 R14: 0000000000000000 R15: 0000000000000000
[  267.673782] [   C22]  </TASK>
[  330.701892] [   C22] rcu: INFO: rcu_sched self-detected stall on CPU
[  330.701895] [   C22] rcu:    22-....: (8402 ticks this GP) idle=fb04/1/0x4000000000000000 softirq=1207/1207 fqs=4193
[  330.701900] [   C22] rcu:    (t=8403 jiffies g=2049 q=2717 ncpus=64)
[  330.701902] [   C22] CPU: 22 UID: 0 PID: 4929 Comm: ls Not tainted 6.12.0release+ #4
[  330.701903] [   C22] Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
[  330.701904] [   C22] RIP: 0010:rhashtable_insert_slow+0x349/0x500
[  330.701907] [   C22] Code: 95 ff 31 c0 eb 10 49 8b 45 30 48 85 c0 49 0f 44 c6 eb 03 4c 89 f0 f0 41 80 24 24 fe f7 44 24 38 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 91 00 00 00 48 3d 00 f0 ff ff 0f 86 15 fd ff ff e9 80 00 00
[  330.701908] [   C22] RSP: 0018:ffff8882155b3870 EFLAGS: 00000286
[  330.701910] [   C22] RAX: ffff88810462d800 RBX: ffff88810e571551 RCX: 0000000000000011
[  330.701911] [   C22] RDX: 0000000000000011 RSI: ffff888259978000 RDI: ffff8882155b3888
[  330.701911] [   C22] RBP: ffff88823cbc7958 R08: 000000008e0492de R09: 0000000000000000
[  330.701912] [   C22] R10: 000000008e0492de R11: ffff88821c326680 R12: ffff88810e571550
[  330.701913] [   C22] R13: ffff88810e571400 R14: fffffffffffffff5 R15: ffff88821c326680
[  330.701913] [   C22] FS:  00007fb9de145740(0000) GS:ffff889fbe580000(0000) knlGS:0000000000000000
[  330.701914] [   C22] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  330.701915] [   C22] CR2: 000055da12573040 CR3: 000000021542d000 CR4: 00000000003506f0
[  330.701916] [   C22] Call Trace:
[  330.701916] [   C22]  <IRQ>
[  330.701917] [   C22]  ? rcu_dump_cpu_stacks+0xdf/0x130
[  330.701919] [   C22]  ? print_cpu_stall+0x173/0x2c0
[  330.701920] [   C22]  ? rcu_sched_clock_irq+0x221/0x5d0
[  330.701921] [   C22]  ? update_process_times+0x71/0xa0
[  330.701923] [   C22]  ? tick_nohz_handler+0xbb/0x110
[  330.701925] [   C22]  ? tick_setup_sched_timer+0x180/0x180
[  330.701926] [   C22]  ? __hrtimer_run_queues+0xf1/0x250
[  330.701927] [   C22]  ? hrtimer_interrupt+0xf0/0x390
[  330.701928] [   C22]  ? __sysvec_apic_timer_interrupt+0x47/0x140
[  330.701930] [   C22]  ? sysvec_apic_timer_interrupt+0x63/0x80
[  330.701931] [   C22]  </IRQ>
[  330.701931] [   C22]  <TASK>
[  330.701931] [   C22]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[  330.701933] [   C22]  ? rhashtable_insert_slow+0x349/0x500
[  330.701934] [   C22]  bch2_inode_hash_insert+0x309/0x480
[  330.701936] [   C22]  bch2_lookup+0x567/0x730
[  330.701938] [   C22]  ? __memcg_slab_post_alloc_hook+0x2ca/0x370
[  330.701940] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x2d/0x1b0
[  330.701941] [   C22]  ? kmem_cache_alloc_lru_noprof+0xe9/0x1b0
[  330.701943] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x14e/0x1b0
[  330.701945] [   C22]  __lookup_slow+0xcb/0x130
[  330.701947] [   C22]  lookup_slow+0x33/0x50
[  330.701948] [   C22]  walk_component+0xcc/0xe0
[  330.701949] [   C22]  path_lookupat+0x4d/0xf0
[  330.701950] [   C22]  filename_lookup+0xa8/0x140
[  330.701952] [   C22]  vfs_statx+0x64/0xf0
[  330.701953] [   C22]  __se_sys_statx+0xc9/0x180
[  330.701954] [   C22]  ? touch_atime+0x23/0x1b0
[  330.701956] [   C22]  ? bch2_vfs_readdir+0xc2/0x160
[  330.701957] [   C22]  ? iterate_dir+0x92/0x120
[  330.701958] [   C22]  ? __se_sys_getdents64+0xa4/0xd0
[  330.701959] [   C22]  ? filldir+0x180/0x180
[  330.701960] [   C22]  do_syscall_64+0x52/0xf0
[  330.701962] [   C22]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  330.701963] [   C22] RIP: 0033:0x7fb9de26884e
[  330.701964] [   C22] Code: 05 0c 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 9a fd ff ff e8 d3 88 01 00 0f 1f 00 f3 0f 1e fa 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2a 89 c1 85 c0 74 0f 48 8b 05 a5 05 0c 00 64
[  330.701965] [   C22] RSP: 002b:00007ffff17b71f8 EFLAGS: 00000246 ORIG_RAX: 000000000000014c
[  330.701966] [   C22] RAX: ffffffffffffffda RBX: 000055da12573be8 RCX: 00007fb9de26884e
[  330.701967] [   C22] RDX: 0000000000000900 RSI: 00007ffff17b7340 RDI: 00000000ffffff9c
[  330.701968] [   C22] RBP: 00007ffff17b7330 R08: 00007ffff17b7210 R09: 00007ffff17b73e0
[  330.701968] [   C22] R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000002
[  330.701969] [   C22] R13: 000055da12573bd0 R14: 0000000000000000 R15: 0000000000000000
[  330.701970] [   C22]  </TASK>
[  393.730278] [   C22] rcu: INFO: rcu_sched self-detected stall on CPU
[  393.730281] [   C22] rcu:    22-....: (14705 ticks this GP) idle=fb04/1/0x4000000000000000 softirq=1207/1207 fqs=7344
[  393.730285] [   C22] rcu:    (t=14706 jiffies g=2049 q=5902 ncpus=64)
[  393.730287] [   C22] CPU: 22 UID: 0 PID: 4929 Comm: ls Not tainted 6.12.0release+ #4
[  393.730289] [   C22] Hardware name: Micro-Star International Co., Ltd. MS-7C59/Creator TRX40 (MS-7C59), BIOS 1.96 04/25/2023
[  393.730289] [   C22] RIP: 0010:rhashtable_insert_slow+0x349/0x500
[  393.730292] [   C22] Code: 95 ff 31 c0 eb 10 49 8b 45 30 48 85 c0 49 0f 44 c6 eb 03 4c 89 f0 f0 41 80 24 24 fe f7 44 24 38 00 02 00 00 74 01 fb 48 85 c0 <0f> 84 91 00 00 00 48 3d 00 f0 ff ff 0f 86 15 fd ff ff e9 80 00 00
[  393.730293] [   C22] RSP: 0018:ffff8882155b3870 EFLAGS: 00000286
[  393.730294] [   C22] RAX: ffff88810462d800 RBX: ffff88810e571551 RCX: 0000000000000011
[  393.730295] [   C22] RDX: 0000000000000011 RSI: ffff888259978000 RDI: ffff8882155b3888
[  393.730296] [   C22] RBP: ffff88823cbc7958 R08: 000000008e0492de R09: 0000000000000000
[  393.730297] [   C22] R10: 000000008e0492de R11: ffff88821c326680 R12: ffff88810e571550
[  393.730297] [   C22] R13: ffff88810e571400 R14: fffffffffffffff5 R15: ffff88821c326680
[  393.730298] [   C22] FS:  00007fb9de145740(0000) GS:ffff889fbe580000(0000) knlGS:0000000000000000
[  393.730299] [   C22] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  393.730300] [   C22] CR2: 000055da12573040 CR3: 000000021542d000 CR4: 00000000003506f0
[  393.730300] [   C22] Call Trace:
[  393.730302] [   C22]  <IRQ>
[  393.730302] [   C22]  ? rcu_dump_cpu_stacks+0xdf/0x130
[  393.730304] [   C22]  ? print_cpu_stall+0x173/0x2c0
[  393.730306] [   C22]  ? rcu_sched_clock_irq+0x221/0x5d0
[  393.730307] [   C22]  ? update_process_times+0x71/0xa0
[  393.730308] [   C22]  ? tick_nohz_handler+0xbb/0x110
[  393.730310] [   C22]  ? tick_setup_sched_timer+0x180/0x180
[  393.730311] [   C22]  ? __hrtimer_run_queues+0xf1/0x250
[  393.730312] [   C22]  ? hrtimer_interrupt+0xf0/0x390
[  393.730313] [   C22]  ? __sysvec_apic_timer_interrupt+0x47/0x140
[  393.730315] [   C22]  ? sysvec_apic_timer_interrupt+0x63/0x80
[  393.730316] [   C22]  </IRQ>
[  393.730316] [   C22]  <TASK>
[  393.730316] [   C22]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[  393.730318] [   C22]  ? rhashtable_insert_slow+0x349/0x500
[  393.730319] [   C22]  bch2_inode_hash_insert+0x309/0x480
[  393.730321] [   C22]  bch2_lookup+0x567/0x730
[  393.730322] [   C22]  ? __memcg_slab_post_alloc_hook+0x2ca/0x370                                                                                                                   [  393.730324] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x2d/0x1b0
[  393.730326] [   C22]  ? kmem_cache_alloc_lru_noprof+0xe9/0x1b0
[  393.730328] [   C22]  ? __d_alloc.llvm.3704585759423112158+0x14e/0x1b0
[  393.730329] [   C22]  __lookup_slow+0xcb/0x130
[  393.730331] [   C22]  lookup_slow+0x33/0x50
[  393.730333] [   C22]  walk_component+0xcc/0xe0
[  393.730334] [   C22]  path_lookupat+0x4d/0xf0
[  393.730335] [   C22]  filename_lookup+0xa8/0x140
[  393.730337] [   C22]  vfs_statx+0x64/0xf0
[  393.730338] [   C22]  __se_sys_statx+0xc9/0x180
[  393.730339] [   C22]  ? touch_atime+0x23/0x1b0
[  393.730340] [   C22]  ? bch2_vfs_readdir+0xc2/0x160
[  393.730342] [   C22]  ? iterate_dir+0x92/0x120
[  393.730343] [   C22]  ? __se_sys_getdents64+0xa4/0xd0
[  393.730344] [   C22]  ? filldir+0x180/0x180
[  393.730345] [   C22]  do_syscall_64+0x52/0xf0
[  393.730346] [   C22]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  393.730347] [   C22] RIP: 0033:0x7fb9de26884e
[  393.730348] [   C22] Code: 05 0c 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 9a fd ff ff e8 d3 88 01 00 0f 1f 00 f3 0f 1e fa 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2a 89 c1 85 c0 74 0f 48 8b 05 a5 05 0c 00 64
[  393.730349] [   C22] RSP: 002b:00007ffff17b71f8 EFLAGS: 00000246 ORIG_RAX: 000000000000014c
[  393.730350] [   C22] RAX: ffffffffffffffda RBX: 000055da12573be8 RCX: 00007fb9de26884e
[  393.730351] [   C22] RDX: 0000000000000900 RSI: 00007ffff17b7340 RDI: 00000000ffffff9c
[  393.730352] [   C22] RBP: 00007ffff17b7330 R08: 00007ffff17b7210 R09: 00007ffff17b73e0
[  393.730352] [   C22] R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000002
[  393.730353] [   C22] R13: 000055da12573bd0 R14: 0000000000000000 R15: 0000000000000000
[  393.730353] [   C22]  </TASK>