Open felixonmars opened 8 months ago
Another server crash from Debian(RevyOS)'s 6.1.61 kernel, but I am not sure if it's the same issue:
[209173.109588] INFO: task node:896412 blocked for more than 124 seconds.
[209173.116301] Not tainted 6.1.61-pisces #2023.12.19.12.48+c60b48221
[209173.123094] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[209173.131065] task:node state:D stack:0 pid:896412 ppid:890296 flags:0x00000100
[209173.139659] Call Trace:
[209173.142239] [<ffffffff80a6e16a>] __schedule+0x29c/0x89e
[209173.147634] [<ffffffff80a6e7b8>] schedule+0x4c/0xce
[209173.152644] [<ffffffff80a6ebba>] schedule_preempt_disabled+0x18/0x20
[209173.159136] [<ffffffff80a6fb82>] __mutex_lock.constprop.0+0x336/0x6b8
[209173.165734] [<ffffffff80a70000>] __mutex_lock_slowpath+0x1a/0x22
[209173.171878] [<ffffffff80a7004a>] mutex_lock+0x42/0x4c
[209173.177061] [<ffffffff800d1a98>] proc_cgroup_show+0x5c/0x3ba
[209173.182866] [<ffffffff80306f7a>] proc_single_show+0x4e/0x9e
[209173.188594] [<ffffffff802a836a>] seq_read_iter+0x158/0x362
[209173.194216] [<ffffffff802a8608>] seq_read+0x94/0xc0
[209173.199231] [<ffffffff802797e2>] vfs_read+0xaa/0x238
[209173.204325] [<ffffffff8027a362>] ksys_read+0x6e/0xe4
[209173.209434] [<ffffffff8027a3f2>] sys_read+0x1a/0x22
[209173.214446] [<ffffffff80003cc8>] ret_from_syscall+0x0/0x2
[209173.229174] INFO: task node:896551 blocked for more than 124 seconds.
[209173.235882] Not tainted 6.1.61-pisces #2023.12.19.12.48+c60b48221
[209173.242680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[209173.250655] task:node state:D stack:0 pid:896551 ppid:889336 flags:0x00000100
[209173.259257] Call Trace:
[209173.261829] [<ffffffff80a6e16a>] __schedule+0x29c/0x89e
[209173.267216] [<ffffffff80a6e7b8>] schedule+0x4c/0xce
[209173.272227] [<ffffffff80a6ebba>] schedule_preempt_disabled+0x18/0x20
[209173.278716] [<ffffffff80a6fb82>] __mutex_lock.constprop.0+0x336/0x6b8
[209173.285303] [<ffffffff80a70000>] __mutex_lock_slowpath+0x1a/0x22
[209173.291479] [<ffffffff80a7004a>] mutex_lock+0x42/0x4c
[209173.296665] [<ffffffff800d1a98>] proc_cgroup_show+0x5c/0x3ba
[209173.302464] [<ffffffff80306f7a>] proc_single_show+0x4e/0x9e
[209173.308205] [<ffffffff802a836a>] seq_read_iter+0x158/0x362
[209173.313822] [<ffffffff802a8608>] seq_read+0x94/0xc0
[209173.318828] [<ffffffff802797e2>] vfs_read+0xaa/0x238
[209173.323937] [<ffffffff8027a362>] ksys_read+0x6e/0xe4
[209173.329031] [<ffffffff8027a3f2>] sys_read+0x1a/0x22
[209173.334039] [<ffffffff80003cc8>] ret_from_syscall+0x0/0x2
[209173.348811] INFO: task node:896742 blocked for more than 124 seconds.
[209173.355521] Not tainted 6.1.61-pisces #2023.12.19.12.48+c60b48221
[209173.362294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[209173.370269] task:node state:D stack:0 pid:896742 ppid:890159 flags:0x00000100
[209173.378867] Call Trace:
[209173.381442] [<ffffffff80a6e16a>] __schedule+0x29c/0x89e
[209173.389371] [<ffffffff80a6e7b8>] schedule+0x4c/0xce
[209173.396810] [<ffffffff80a6ebba>] schedule_preempt_disabled+0x18/0x20
[209173.405501] [<ffffffff80a6fb82>] __mutex_lock.constprop.0+0x336/0x6b8
[209173.414293] [<ffffffff80a70000>] __mutex_lock_slowpath+0x1a/0x22
[209173.422578] [<ffffffff80a7004a>] mutex_lock+0x42/0x4c
[209173.429832] [<ffffffff800d1a98>] proc_cgroup_show+0x5c/0x3ba
[209173.437697] [<ffffffff80306f7a>] proc_single_show+0x4e/0x9e
[209173.445493] [<ffffffff802a836a>] seq_read_iter+0x158/0x362
[209173.453113] [<ffffffff802a8608>] seq_read+0x94/0xc0
[209173.460196] [<ffffffff802797e2>] vfs_read+0xaa/0x238
[209173.467368] [<ffffffff8027a362>] ksys_read+0x6e/0xe4
[209173.474484] [<ffffffff8027a3f2>] sys_read+0x1a/0x22
[209173.481473] [<ffffffff80003cc8>] ret_from_syscall+0x0/0x2
sbi_trap_error: hart93: illegal instruction handler failed (error -2)
sbi_trap_error: hart93: mcause=0x0000000000000002 mtval=0x0000000000000000
sbi_trap_error: hart93: mepc=0x000000000015b1fa mstatus=0x0000000a00001820
sbi_trap_error: hart93: ra=0x000000000015b1f8 sp=0x00000000000e1f18
sbi_trap_error: hart93: gp=0xffffffff81a44ec8 tp=0xffffffe2b2a78000
sbi_trap_error: hart93: s0=0x00000000000e1f38 s1=0xffffffe81d005400
sbi_trap_error: hart93: a0=0x0013000000130000 a1=0xffffffffb51fb51f
sbi_trap_error: hart93: a2=0x000000000000b51f a3=0x000000000000b51f
sbi_trap_error: hart93: a4=0x000000000000b520 a5=0x0013000000130000
sbi_trap_error: hart93: a6=0x000000000000b51f a7=0x0000000000000080
sbi_trap_error: hart93: s2=0xfffffff65fa5f180 s3=0x0000000000016b20
sbi_trap_error: hart93: s4=0x0000000000000009 s5=0x0000000000000009
sbi_trap_error: hart93: s6=0x0000000200000022 s7=0xfffffff65fa5f8ff
sbi_trap_error: hart93: s8=0xffffffff81acb488 s9=0x000000000000005d
sbi_trap_error: hart93: s10=0xffffffff81a6ac98 s11=0xffffffe04f79bc08
sbi_trap_error: hart93: t0=0x0000000a00000820 t1=0x0000000000000001
sbi_trap_error: hart93: t2=0xffffffff8100bf28 t3=0x00000000ff00ff00
sbi_trap_error: hart93: t4=0xffffffd8ffa8fd38 t5=0x0000000000000002
sbi_trap_error: hart93: t6=0x02b11ee2af76cd48
Another crash with the first self-compiled kernel, on the same host:
[53111.356078] nvme nvme0: Abort status: 0x0
[53111.370408] nvme nvme0: I/O 309 (Write) QID 1 timeout, aborting
[53111.374461] nvme nvme0: Abort status: 0x0
[53111.388915] nvme nvme0: I/O 310 (Write) QID 1 timeout, aborting
[53111.392554] nvme nvme0: Abort status: 0x0
[53111.406580] nvme nvme0: I/O 311 (Write) QID 1 timeout, aborting
[53111.410602] nvme nvme0: Abort status: 0x0
[53111.424884] nvme nvme0: I/O 312 (Write) QID 1 timeout, aborting
[53111.428874] nvme nvme0: Abort status: 0x0
[53111.443731] nvme nvme0: Abort status: 0x0
[55284.493268] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[55284.503385] rcu: 41-...0: (1 GPs behind) idle=7b24/1/0x4000000000000000 softirq=3305695/3305728 fqs=1979
[55312.513142] watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [migration/49:314]
[55312.525061] Modules linked in: tun cfg80211 rfkill xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_TCPMSS xt_tcpudp iptable_filter vfat fat ixgbe mdio_devres ofpart of_mdio ipmi_si fixed_phy sophgo_spifmc fwnode_mdio spi_nor ipmi_devintf libphy igb 8250_dw gpio_dwapb ipmi_msghandler mtd mdio switchtec mousedev uio_pdrv_genirq uio tcp_bbr sch_fq fuse loop dm_mod nfnetlink bpf_preload ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 mmc_block sdhci_sophgo ast sdhci_pltfm usbhid nvme drm_vram_helper sdhci spi_dw_mmio drm_ttm_helper gpio_keys mmc_core spi_dw nvme_core ttm xhci_pci nvme_common xhci_pci_renesas
[55312.607363] CPU: 49 PID: 314 Comm: migration/49 Not tainted 6.1.61-1.1-sophgo-multi-08348-gdb74e759247f #1 d71360b18e23f727e028cd5c570bc8e9bdca43d0
[55312.628591] Hardware name: Sophgo Mango (DT)
[55312.636471] Stopper: multi_cpu_stop+0x0/0x172 <- migrate_swap+0xbe/0x158
[55312.647469] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[55312.656635] ra : multi_cpu_stop+0xb8/0x172
[55312.664391] epc : ffffffff8009ab24 ra : ffffffff800ef2cc sp : ffffffc80b0fbd90
[55312.675607] gp : ffffffff81bac808 tp : ffffffe7fed63f00 t0 : 0000000000000080
[55312.686709] t1 : 0000000001806000 t2 : 0000000000000001 s0 : ffffffc80b0fbe10
[55312.697704] s1 : ffffffc8390eb970 a0 : ffffffff80e0da88 a1 : 0000000000000002
[55312.708524] a2 : ffffffc8390eb9c8 a3 : ffffffff81c089c8 a4 : ffffffffe72fec6c
[55312.719750] a5 : fffffff5db55bcc0 a6 : 0000000000000001 a7 : 0000000000000000
[55312.730866] s2 : ffffffc8390eb994 s3 : ffffffffffffffff s4 : ffffffff80e0da88
[55312.741914] s5 : 0000000000000001 s6 : 0000000000000000 s7 : 0000000000000002
[55312.752799] s8 : 0000000000000004 s9 : 0000000000000001 s10: 0000000000000003
[55312.763432] s11: 0000000000000001 t3 : 0000000000000000 t4 : 0000000000000000
[55312.774591] t5 : 0000000000000034 t6 : 000000000000ffff
[55312.783551] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000001
[55312.794943] [<ffffffff8009ab24>] rcu_momentary_dyntick_idle+0x3a/0x80
[55312.804845] [<ffffffff800eee3a>] cpu_stopper_thread+0xfc/0x182
[55312.813935] [<ffffffff80044e76>] smpboot_thread_fn+0xe6/0x11a
[55312.823287] [<ffffffff80040aee>] kthread+0xbe/0xd4
[55312.831636] [<ffffffff80003f18>] ret_from_exception+0x0/0x16
[55340.510039] watchdog: BUG: soft lockup - CPU#49 stuck for 48s! [migration/49:314]
[55340.521060] Modules linked in: tun cfg80211 rfkill xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_TCPMSS xt_tcpudp iptable_filter vfat fat ixgbe mdio_devres ofpart of_mdio ipmi_si fixed_phy sophgo_spifmc fwnode_mdio spi_nor ipmi_devintf libphy igb 8250_dw gpio_dwapb ipmi_msghandler mtd mdio switchtec mousedev uio_pdrv_genirq uio tcp_bbr sch_fq fuse loop dm_mod nfnetlink bpf_preload ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 mmc_block sdhci_sophgo ast sdhci_pltfm usbhid nvme drm_vram_helper sdhci spi_dw_mmio drm_ttm_helper gpio_keys mmc_core spi_dw nvme_core ttm xhci_pci nvme_common xhci_pci_renesas
[55340.599220] CPU: 49 PID: 314 Comm: migration/49 Tainted: G L 6.1.61-1.1-sophgo-multi-08348-gdb74e759247f #1 d71360b18e23f727e028cd5c570bc8e9bdca43d0
[55340.619981] Hardware name: Sophgo Mango (DT)
[55340.627532] Stopper: multi_cpu_stop+0x0/0x172 <- migrate_swap+0xbe/0x158
[55340.637316] epc : rcu_momentary_dyntick_idle+0x3a/0x80
[55340.645582] ra : multi_cpu_stop+0xb8/0x172
[55340.652699] epc : ffffffff8009ab24 ra : ffffffff800ef2cc sp : ffffffc80b0fbd90
[55340.663138] gp : ffffffff81bac808 tp : ffffffe7fed63f00 t0 : 0000000000000080
[55340.673447] t1 : 0000000001806000 t2 : 0000000000000001 s0 : ffffffc80b0fbe10
[55340.683806] s1 : ffffffc8390eb970 a0 : ffffffff80e0da88 a1 : 0000000000000002
[55340.694164] a2 : ffffffc8390eb9c8 a3 : ffffffff81c089c8 a4 : ffffffffc2766c74
[55340.704272] a5 : fffffff5db55bcc0 a6 : 0000000000000001 a7 : 0000000000000000
[55340.714400] s2 : ffffffc8390eb994 s3 : ffffffffffffffff s4 : ffffffff80e0da88
[55340.724650] s5 : 0000000000000001 s6 : 0000000000000000 s7 : 0000000000000002
[55340.734871] s8 : 0000000000000004 s9 : 0000000000000001 s10: 0000000000000003
[55340.745127] s11: 0000000000000001 t3 : 0000000000000000 t4 : 0000000000000000
[55340.755163] t5 : 0000000000000034 t6 : 000000000000ffff
[55340.763195] status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000001
[55340.774063] [<ffffffff8009ab24>] rcu_momentary_dyntick_idle+0x3a/0x80
[55340.783460] [<ffffffff800eee3a>] cpu_stopper_thread+0xfc/0x182
[55340.792051] [<ffffffff80044e76>] smpboot_thread_fn+0xe6/0x11a
[55340.800608] [<ffffffff80040aee>] kthread+0xbe/0xd4
[55340.808312] [<ffffffff80003f18>] ret_from_exception+0x0/0x16
I am getting another 2-way server crash with 6.1.61 kernel compiled myself at commit db74e759247f. The server was having high load (~500) at the moment.