search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
890 stars 336 forks source link

Soft lockup on arm64 v0.96 #513

Open arter97 opened 1 year ago

arter97 commented 1 year ago

Hi, everyone.

I'm hitting soft lockup issues on mptcp_v0.96 on an arm64 server, using shadowsocks-rust.

The issue happens randomly - sometimes once a few weeks, sometimes less than a single day of uptime.

The debugging is limited due to kernel completely locking itself up, but systemd managed to save the following bits:

May  6 16:04:52 arter97-ampere kernel: [1950069.732944] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:04:52 arter97-ampere kernel: [1950069.744985] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:04:52 arter97-ampere kernel: [1950069.745058] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O      5.4.240-arter97+ #1
May  6 16:04:52 arter97-ampere kernel: [1950069.745058] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:04:52 arter97-ampere kernel: [1950069.745059] pstate: 00400005 (nzcv daif +PAN -UAO)
May  6 16:04:52 arter97-ampere kernel: [1950069.745062] pc : mptcp_data_ready+0x5c/0x1ad8
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] lr : mptcp_data_ready+0x1790/0x1ad8
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] sp : ffffff85d0297970
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] x29: ffffff85d0297ff0 x28: ffffff85ce6040c0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745064] x27: ffffff85d5a49bc0 x26: ffffff84ab19ab28 
May  6 16:04:52 arter97-ampere kernel: [1950069.745065] x25: ffffff85d5a49c90 x24: ffffff84ab19aa00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745065] x23: ffffff85ce6040c0 x22: ffffff84ab19aa00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745066] x21: 00000000253f0aac x20: ffffff85fa11dc00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745066] x19: ffffff85d5a49c90 x18: 0000000000000000 
May  6 16:04:52 arter97-ampere kernel: [1950069.745067] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745067] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:04:52 arter97-ampere kernel: [1950069.745068] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:04:52 arter97-ampere kernel: [1950069.745068] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:04:52 arter97-ampere kernel: [1950069.745069] x9 : 00000000253f0aac x8 : 000000000000004e 
May  6 16:04:52 arter97-ampere kernel: [1950069.745069] x7 : 0000000000000080 x6 : 00000000b4792267 
May  6 16:04:52 arter97-ampere kernel: [1950069.745070] x5 : ffffff84ab19a100 x4 : ffffff85fffa05a0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745070] x3 : 00000000a0a77e4d x2 : 0000000000000000 
May  6 16:04:52 arter97-ampere kernel: [1950069.745071] x1 : 0000000000000032 x0 : 0000000000000002 
May  6 16:04:52 arter97-ampere kernel: [1950069.745072] Call trace:
May  6 16:04:52 arter97-ampere kernel: [1950069.745072]  mptcp_data_ready+0x5c/0x1ad8
May  6 16:06:12 arter97-ampere kernel: [1950149.972522] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:06:12 arter97-ampere kernel: [1950149.974950] rcu:    1-....: (60333 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=29993 
May  6 16:06:12 arter97-ampere kernel: [1950149.975757]     (t=60003 jiffies g=39741721 q=1720)
May  6 16:06:12 arter97-ampere kernel: [1950149.975758] Task dump for CPU 1:
May  6 16:06:12 arter97-ampere kernel: [1950149.975759] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:06:12 arter97-ampere kernel: [1950149.975761] Call trace:
May  6 16:06:12 arter97-ampere kernel: [1950149.975765]  dump_backtrace+0x0/0x160
May  6 16:06:12 arter97-ampere kernel: [1950149.975767]  sched_show_task+0x110/0x13c
May  6 16:06:12 arter97-ampere kernel: [1950149.975770]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:06:12 arter97-ampere kernel: [1950149.975777]  kfree+0x7c/0x238
May  6 16:08:04 arter97-ampere kernel: [1950261.731939] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:08:04 arter97-ampere kernel: [1950261.733349] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:08:04 arter97-ampere kernel: [1950261.733377] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:08:04 arter97-ampere kernel: [1950261.733378] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:08:04 arter97-ampere kernel: [1950261.733378] pstate: a0400005 (NzCv daif +PAN -UAO)
May  6 16:08:04 arter97-ampere kernel: [1950261.733382] pc : __kmalloc_track_caller+0xc4/0x280
May  6 16:08:04 arter97-ampere kernel: [1950261.733383] lr : __kmalloc_track_caller+0x4c/0x280
May  6 16:08:04 arter97-ampere kernel: [1950261.733384] sp : ffffff85d02978d0
May  6 16:08:04 arter97-ampere kernel: [1950261.733384] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:08:04 arter97-ampere kernel: [1950261.733385] x27: ffffff85d5a49bc0 x26: ffffffc0111aa300 
May  6 16:08:04 arter97-ampere kernel: [1950261.733385] x25: 0000000000000a20 x24: ffffff84ab19aa00 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x23: ffffffc0111aa300 x22: 0000000000082a20 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x21: 0000000000000180 x20: ffffff85f1b6b800 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x19: ffffff85f8b42c00 x18: 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733387] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:08:04 arter97-ampere kernel: [1950261.733387] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:08:04 arter97-ampere kernel: [1950261.733388] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:08:04 arter97-ampere kernel: [1950261.733388] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:08:04 arter97-ampere kernel: [1950261.733389] x9 : ffffff84c02b3680 x8 : 0000000000010000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733389] x7 : 0000000000000080 x6 : 0000000136ab2517 
May  6 16:08:04 arter97-ampere kernel: [1950261.733390] x5 : 000000010b7ef967 x4 : ffffff85fff9efa0 
May  6 16:08:04 arter97-ampere kernel: [1950261.733391] x3 : 000000010b7ef968 x2 : 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733391] x1 : 0000000000000000 x0 : 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733392] Call trace:
May  6 16:08:04 arter97-ampere kernel: [1950261.733393]  __kmalloc_track_caller+0xc4/0x280
May  6 16:09:13 arter97-ampere kernel: [1950329.983581] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:09:13 arter97-ampere kernel: [1950329.984740] rcu:    1-....: (105336 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=52494 
May  6 16:09:13 arter97-ampere kernel: [1950329.985548]     (t=105006 jiffies g=39741721 q=2651)
May  6 16:09:13 arter97-ampere kernel: [1950329.985549] Task dump for CPU 1:
May  6 16:09:13 arter97-ampere kernel: [1950329.985549] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:09:13 arter97-ampere kernel: [1950329.985551] Call trace:
May  6 16:09:13 arter97-ampere kernel: [1950329.985555]  dump_backtrace+0x0/0x160
May  6 16:09:13 arter97-ampere kernel: [1950329.985558]  sched_show_task+0x110/0x13c
May  6 16:09:13 arter97-ampere kernel: [1950329.985561]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:09:13 arter97-ampere kernel: [1950329.985562]  skb_split+0x1ac/0x2b4
May  6 16:11:00 arter97-ampere kernel: [1950437.731020] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:11:00 arter97-ampere kernel: [1950437.732698] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:11:00 arter97-ampere kernel: [1950437.732725] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:11:00 arter97-ampere kernel: [1950437.732725] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:11:00 arter97-ampere kernel: [1950437.732726] pstate: 40400005 (nZcv daif +PAN -UAO)
May  6 16:11:00 arter97-ampere kernel: [1950437.732730] pc : kfree+0x44/0x238
May  6 16:11:00 arter97-ampere kernel: [1950437.732732] lr : __kfree_skb+0x1c/0xc0
May  6 16:11:00 arter97-ampere kernel: [1950437.732732] sp : ffffff85d0297940
May  6 16:11:00 arter97-ampere kernel: [1950437.732733] x29: ffffff85d0297ff0 x28: ffffff85ce6040c0 
May  6 16:11:00 arter97-ampere kernel: [1950437.732733] x27: ffffff85d5a49bc0 x26: ffffff84ab19aa28 
May  6 16:11:00 arter97-ampere kernel: [1950437.732734] x25: ffffff85d5a49c90 x24: ffffff84ab19aa00 
May  6 16:11:00 arter97-ampere kernel: [1950437.732734] x23: ffffff85ce6040c0 x22: ffffff84ab19ab00 
May  6 16:11:00 arter97-ampere kernel: [1950437.732735] x21: ffffffff17a6da80 x20: ffffff85f1b6a600 
May  6 16:11:00 arter97-ampere kernel: [1950437.732735] x19: ffffff84ab19aa00 x18: 0000000000000000 
May  6 16:11:00 arter97-ampere kernel: [1950437.732736] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:11:00 arter97-ampere kernel: [1950437.732736] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:11:00 arter97-ampere kernel: [1950437.732737] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:11:00 arter97-ampere kernel: [1950437.732737] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:11:00 arter97-ampere kernel: [1950437.732738] x9 : 00000000253f0aac x8 : 000000000000004e 
May  6 16:11:00 arter97-ampere kernel: [1950437.732738] x7 : 0000000000000080 x6 : 0000000000000006 
May  6 16:11:00 arter97-ampere kernel: [1950437.732739] x5 : ffffff84ab19ab00 x4 : 0000000000008a34 
May  6 16:11:00 arter97-ampere kernel: [1950437.732739] x3 : 00000000001fffff x2 : 0000000000000000 
May  6 16:11:00 arter97-ampere kernel: [1950437.732740] x1 : 0000000000000000 x0 : ffffffff17a51e80 
May  6 16:11:00 arter97-ampere kernel: [1950437.732741] Call trace:
May  6 16:11:00 arter97-ampere kernel: [1950437.732742]  kfree+0x44/0x238
May  6 16:12:13 arter97-ampere kernel: [1950509.994642] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:12:13 arter97-ampere kernel: [1950509.998005] rcu:    1-....: (150339 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=74995 
May  6 16:12:13 arter97-ampere kernel: [1950509.999143]     (t=150009 jiffies g=39741721 q=3325)
May  6 16:12:13 arter97-ampere kernel: [1950509.999143] Task dump for CPU 1:
May  6 16:12:13 arter97-ampere kernel: [1950509.999144] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:12:13 arter97-ampere kernel: [1950509.999146] Call trace:
May  6 16:12:13 arter97-ampere kernel: [1950509.999150]  dump_backtrace+0x0/0x160
May  6 16:12:13 arter97-ampere kernel: [1950509.999152]  sched_show_task+0x110/0x13c
May  6 16:12:13 arter97-ampere kernel: [1950509.999155]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:12:13 arter97-ampere kernel: [1950509.999156]  mptcp_data_ready+0x550/0x1ad8
May  6 16:13:56 arter97-ampere kernel: [1950613.730100] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:13:56 arter97-ampere kernel: [1950613.734286] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:13:56 arter97-ampere kernel: [1950613.734312] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:13:56 arter97-ampere kernel: [1950613.734312] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:13:56 arter97-ampere kernel: [1950613.734313] pstate: 00400005 (nzcv daif +PAN -UAO)
May  6 16:13:56 arter97-ampere kernel: [1950613.734317] pc : skb_split+0x4c/0x2b4
May  6 16:13:56 arter97-ampere kernel: [1950613.734319] lr : mptcp_data_ready+0xed0/0x1ad8
May  6 16:13:56 arter97-ampere kernel: [1950613.734319] sp : ffffff85d0297930
May  6 16:13:56 arter97-ampere kernel: [1950613.734320] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:13:56 arter97-ampere kernel: [1950613.734320] x27: ffffff85d5a49bc0 x26: ffffff84ab19ab28 
May  6 16:13:56 arter97-ampere kernel: [1950613.734321] x25: ffffff85d5a49c90 x24: 0000000000010000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734321] x23: ffffff85ce6040c0 x22: 0000000000010000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734322] x21: 0000000000000000 x20: ffffff84ab19aa00 
May  6 16:13:56 arter97-ampere kernel: [1950613.734322] x19: ffffff84ab19ab00 x18: 0000000000000000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734323] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734323] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:13:56 arter97-ampere kernel: [1950613.734324] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:13:56 arter97-ampere kernel: [1950613.734324] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x9 : 0000000000000000 x8 : 0000000000000300 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x7 : 0000000000000000 x6 : 00000000000000c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x5 : 0000000000000000 x4 : ffffff85f1b6a6c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734326] x3 : 0000000000000000 x2 : 0000000000000000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734327] x1 : ffffff84ab19ab00 x0 : ffffff84ab19aa00 
May  6 16:13:56 arter97-ampere kernel: [1950613.734327] Call trace:
May  6 16:13:56 arter97-ampere kernel: [1950613.734328]  skb_split+0x4c/0x2b4
May  6 16:15:13 arter97-ampere kernel: [1950690.005700] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:15:13 arter97-ampere kernel: [1950690.013846] rcu:    1-....: (195341 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=97495 
May  6 16:15:13 arter97-ampere kernel: [1950690.014720]     (t=195012 jiffies g=39741721 q=3771)
May  6 16:15:13 arter97-ampere kernel: [1950690.014721] Task dump for CPU 1:
May  6 16:15:13 arter97-ampere kernel: [1950690.014722] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:15:13 arter97-ampere kernel: [1950690.014724] Call trace:
May  6 16:15:13 arter97-ampere kernel: [1950690.014728]  dump_backtrace+0x0/0x160
May  6 16:15:13 arter97-ampere kernel: [1950690.014730]  sched_show_task+0x110/0x13c
May  6 16:15:13 arter97-ampere kernel: [1950690.014732]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:15:13 arter97-ampere kernel: [1950690.014734]  skb_split+0x19c/0x2b4
May  6 16:16:52 arter97-ampere kernel: [1950789.729181] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:16:52 arter97-ampere kernel: [1950789.738604] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:16:52 arter97-ampere kernel: [1950789.738660] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:16:52 arter97-ampere kernel: [1950789.738660] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:16:52 arter97-ampere kernel: [1950789.738661] pstate: a0400005 (NzCv daif +PAN -UAO)
May  6 16:16:52 arter97-ampere kernel: [1950789.738666] pc : __kmalloc_track_caller+0xc4/0x280
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] lr : __kmalloc_track_caller+0x4c/0x280
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] sp : ffffff85d02978d0
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:16:52 arter97-ampere kernel: [1950789.738668] x27: ffffff85d5a49bc0 x26: ffffffc0111aa300 
May  6 16:16:52 arter97-ampere kernel: [1950789.738668] x25: 0000000000000a20 x24: ffffff84ab19aa00 
May  6 16:16:52 arter97-ampere kernel: [1950789.738669] x23: ffffffc0111aa300 x22: 0000000000082a20 
May  6 16:16:52 arter97-ampere kernel: [1950789.738669] x21: 0000000000000180 x20: ffffff85f1b6b800 
May  6 16:16:52 arter97-ampere kernel: [1950789.738670] x19: ffffff85f8b42c00 x18: 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738670] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:16:52 arter97-ampere kernel: [1950789.738671] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:16:52 arter97-ampere kernel: [1950789.738671] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:16:52 arter97-ampere kernel: [1950789.738672] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:16:52 arter97-ampere kernel: [1950789.738672] x9 : ffffff84c02b3680 x8 : 0000000000010000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738673] x7 : 0000000000000080 x6 : 000000029cb76c4f 
May  6 16:16:52 arter97-ampere kernel: [1950789.738673] x5 : 00000002718b409f x4 : ffffff85fff9efa0 
May  6 16:16:52 arter97-ampere kernel: [1950789.738674] x3 : 00000002718b40a0 x2 : 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738674] x1 : 0000000000000000 x0 : 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738675] Call trace:
May  6 16:16:52 arter97-ampere kernel: [1950789.738676]  __kmalloc_track_caller+0xc4/0x280
May  6 16:18:13 arter97-ampere kernel: [1950870.016760] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:18:13 arter97-ampere kernel: [1950870.027787] rcu:    1-....: (240341 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=119993 
May  6 16:18:13 arter97-ampere kernel: [1950870.028576]     (t=240015 jiffies g=39741721 q=4222)
May  6 16:18:13 arter97-ampere kernel: [1950870.028577] Task dump for CPU 1:
May  6 16:18:13 arter97-ampere kernel: [1950870.028578] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:18:13 arter97-ampere kernel: [1950870.028582] Call trace:
May  6 16:18:13 arter97-ampere kernel: [1950870.028586]  dump_backtrace+0x0/0x160
May  6 16:18:13 arter97-ampere kernel: [1950870.028588]  sched_show_task+0x110/0x13c
May  6 16:18:13 arter97-ampere kernel: [1950870.028591]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:18:13 arter97-ampere kernel: [1950870.028595]  __kmalloc_track_caller+0xc4/0x280

This is the MPTCP's configuration:

root@arter97-ampere:/proc/sys/net/mptcp# grep . *
mptcp_checksum:1
mptcp_debug:0
mptcp_enabled:1
mptcp_path_manager:fullmesh
mptcp_scheduler:default
mptcp_syn_retries:3
mptcp_version:0

Thanks in advance.

matttbe commented 1 year ago

Hi,

Thank you for the bug report.

Sorry, I was off and I missed the notification.

Do you mind executing ./scripts/decode_stacktrace.sh on the stacktrace you shared (and maybe share the source code if the lines doesn't correspond to what we have here)

I have to admit that there is not so much info in the stacktrace (maybe there are debug kernel options that could help?) + no easy way to reproduce it, it doesn't look good :-/

arter97 commented 1 year ago

Hi @matttbe, I've managed to capture a decoded stacktrace.

[20299.002613] rcu: INFO: rcu_sched self-detected stall on CPU
[20299.023487] rcu:      1-....: (15169 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=7191
[20299.027940]   (t=15007 jiffies g=266325 q=301)
[20299.027941] Task dump for CPU 1:
[20299.027942] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20299.027944] Call trace:
[20299.027968] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20299.027974] sched_show_task (kernel/sched/core.c:6101)
[20299.027976] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20299.027980] skb_split (net/core/skbuff.c:3242)
[20399.922033] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20399.924383] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20399.924495] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O      5.4.244-arter97+ #1
[20399.924495] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20399.924496] pstate: 80400005 (Nzcv daif +PAN -UAO)
[20399.924500] pc : skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20399.924500] lr : skb_split (net/core/skbuff.c:3286)
[20399.924501] sp : ffffff85cf35f940
[20399.924501] x29: ffffff85cf35fff0 x28: 0000000000000034
[20399.924502] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20399.924502] x25: ffffff85d6719c90 x24: 0000000000010000
[20399.924503] x23: 0000000000000000 x22: 0000000000000000
[20399.924503] x21: 0000000000000000 x20: ffffff85a1daf300
[20399.924504] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20399.924504] x17: 0000000000000001 x16: ffffffc010edad60
[20399.924505] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20399.924505] x13: 000039ce00003cc0 x12: ffffffff16262400
[20399.924506] x11: 00000000038382bf x10: 00000000000000c0
[20399.924511] x9 : 0000000000000001 x8 : 0000000000000008
[20399.924512] x7 : 0000000000000040 x6 : 0000000000000001
[20399.924512] x5 : 0000000000000972 x4 : 0000000000000001
[20399.924513] x3 : 0000000000003096 x2 : ffffffff16177a00
[20399.924513] x1 : ffffff85ccc62ec0 x0 : 0000000000003a08
[20399.924514] Call trace:
[20399.924515] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20479.013589] rcu: INFO: rcu_sched self-detected stall on CPU
[20479.025167] rcu:      1-....: (60167 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=28163
[20479.026135]   (t=60007 jiffies g=266325 q=1128)
[20479.026136] Task dump for CPU 1:
[20479.026137] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20479.026139] Call trace:
[20479.026143] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20479.026145] sched_show_task (kernel/sched/core.c:6101)
[20479.026147] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20479.026156] kfree (mm/slub.c:4018 (discriminator 2))
[20575.921070] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20575.935785] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20575.935941] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20575.935941] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20575.935942] pstate: 80400005 (Nzcv daif +PAN -UAO)
[20575.935946] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:932 (discriminator 1))
[20575.935947] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[20575.935948] sp : ffffff85cf35f980
[20575.935948] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[20575.935949] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[20575.935949] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[20575.935950] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[20575.935950] x21: 000000007cd21279 x20: ffffff85a1daf300
[20575.935951] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[20575.935951] x17: 0000000000000001 x16: ffffffc010edad60
[20575.935952] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20575.935952] x13: 000039ce00003cc0 x12: ffffffff16262400
[20575.935953] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[20575.935953] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[20575.935954] x7 : 00000000000000a0 x6 : 0000000103eaed34
[20575.935954] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[20575.935955] x3 : 000000007cd21279 x2 : ffffff85d1b5f080
[20575.935955] x1 : 0000000000000000 x0 : 000000007cd31279
[20575.935956] Call trace:
[20575.935957] mptcp_data_ready (net/mptcp/mptcp_input.c:932 (discriminator 1))
[20659.024622] rcu: INFO: rcu_sched self-detected stall on CPU
[20659.026880] rcu:      1-....: (105165 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=48876
[20659.027610]   (t=105007 jiffies g=266325 q=1683)
[20659.027610] Task dump for CPU 1:
[20659.027611] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20659.027613] Call trace:
[20659.027617] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20659.027620] sched_show_task (kernel/sched/core.c:6101)
[20659.027622] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20659.027624] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20767.920032] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20767.929994] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20767.934512] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20767.934516] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20767.934517] pstate: 40400005 (nZcv daif +PAN -UAO)
[20767.934531] pc : sock_rfree (include/net/sock.h:1469 (discriminator 1))
[20767.934532] lr : skb_release_head_state (include/linux/skbuff.h:4090)
[20767.934532] sp : ffffff85cf35f960
[20767.934533] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[20767.934533] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20767.934534] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[20767.934534] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[20767.934535] x21: 000000000383f80f x20: ffffff85d31ed400
[20767.934535] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20767.934536] x17: 0000000000000001 x16: ffffffc010edad60
[20767.934536] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20767.934537] x13: 000039ce00003cc0 x12: ffffffff16262400
[20767.934537] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[20767.934538] x9 : 000000000383f80f x8 : 000000000000004e
[20767.934538] x7 : 00000000000000a0 x6 : 0000000000000008
[20767.934539] x5 : ffffff85a1daf300 x4 : fe0fcb6f0383f80f
[20767.934540] x3 : ffffffc011aa6380 x2 : ffffffc0119d7238
[20767.934540] x1 : 0000000000000300 x0 : ffffff85d6719bc0
[20767.934541] Call trace:
[20767.934542] sock_rfree (include/net/sock.h:1469 (discriminator 1))
[20839.035645] rcu: INFO: rcu_sched self-detected stall on CPU
[20839.037368] rcu:      1-....: (150165 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=68832
[20839.038103]   (t=150010 jiffies g=266325 q=2155)
[20839.038103] Task dump for CPU 1:
[20839.038104] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20839.038106] Call trace:
[20839.038110] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20839.038116] sched_show_task (kernel/sched/core.c:6101)
[20839.038118] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20839.038123] __kmalloc_track_caller (arch/arm64/include/asm/atomic_lse.h:317)
[20943.919071] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20943.920924] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20943.920955] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20943.920956] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20943.920957] pstate: 20400005 (nzCv daif +PAN -UAO)
[20943.920960] pc : skb_split (net/core/skbuff.c:3247)
[20943.920961] lr : skb_split (net/core/skbuff.c:3286)
[20943.920961] sp : ffffff85cf35f940
[20943.920962] x29: ffffff85cf35fff0 x28: 0000000000000034
[20943.920962] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20943.920963] x25: ffffff85d6719c90 x24: 0000000000010000
[20943.920963] x23: 0000000000000000 x22: 0000000000000000
[20943.920964] x21: 0000000000000000 x20: ffffff85a1daf300
[20943.920964] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20943.920965] x17: 0000000000000001 x16: ffffffc010edad60
[20943.920965] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20943.920966] x13: 000039ce00003cc0 x12: ffffffff16262400
[20943.920966] x11: 00000000038382bf x10: 00000000000000c0
[20943.920967] x9 : 0000000000000001 x8 : 0000000000000008
[20943.920967] x7 : 0000000000000090 x6 : 0000000000000006
[20943.920968] x5 : 000000000000c1c8 x4 : 0000000000000006
[20943.920968] x3 : ffffff85ccc62f50 x2 : 0000000000002232
[20943.920969] x1 : ffffff85ccc62ec0 x0 : 000000000000c1c8
[20943.920970] Call trace:
[20943.920970] skb_split (net/core/skbuff.c:3247)
[21019.046661] rcu: INFO: rcu_sched self-detected stall on CPU
[21019.048544] rcu:      1-....: (195168 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=88349
[21019.049660]   (t=195013 jiffies g=266325 q=2419)
[21019.049661] Task dump for CPU 1:
[21019.049662] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21019.049664] Call trace:
[21019.049668] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21019.049670] sched_show_task (kernel/sched/core.c:6101)
[21019.049672] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21019.049674] mptcp_data_ready (net/mptcp/mptcp_input.c:689)
[21119.918105] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[21119.920736] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21119.920778] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21119.920779] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21119.920780] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21119.920783] pc : skb_split (net/core/skbuff.c:3248 (discriminator 2))
[21119.920784] lr : skb_split (net/core/skbuff.c:3286)
[21119.920784] sp : ffffff85cf35f940
[21119.920784] x29: ffffff85cf35fff0 x28: 0000000000000034
[21119.920785] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21119.920786] x25: ffffff85d6719c90 x24: 0000000000010000
[21119.920786] x23: 0000000000000000 x22: 0000000000000000
[21119.920787] x21: 0000000000000000 x20: ffffff85a1dafd00
[21119.920787] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[21119.920787] x17: 0000000000000001 x16: ffffffc010edad60
[21119.920788] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21119.920788] x13: 000039ce00003cc0 x12: ffffffff16262400
[21119.920789] x11: 00000000038382bf x10: 00000000000000c0
[21119.920789] x9 : 0000000000000001 x8 : 0000000000000008
[21119.920790] x7 : 0000000000000060 x6 : 0000000000000003
[21119.920790] x5 : 000000000000803a x4 : 0000000000000003
[21119.920791] x3 : ffffff85ccc63520 x2 : 0000000000000356
[21119.920791] x1 : ffffff85ccc634c0 x0 : 0000000000008390
[21119.920792] Call trace:
[21119.920793] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[21199.057667] rcu: INFO: rcu_sched self-detected stall on CPU
[21199.061108] rcu:      1-....: (240171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=108570
[21199.066967]   (t=240018 jiffies g=266325 q=2750)
[21199.066967] Task dump for CPU 1:
[21199.066969] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21199.066971] Call trace:
[21199.066975] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21199.066977] sched_show_task (kernel/sched/core.c:6101)
[21199.066979] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21199.066981] kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[21295.917138] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21295.922540] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21295.922568] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21295.922569] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21295.922570] pstate: 00400005 (nzcv daif +PAN -UAO)
[21295.922573] pc : skb_split (net/core/skbuff.c:3280)
[21295.922576] lr : mptcp_data_ready (include/linux/skbuff.h:1988)
[21295.922577] sp : ffffff85cf35f940
[21295.922577] x29: ffffff85cf35fff0 x28: 0000000000000034
[21295.922577] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21295.922578] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21295.922578] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21295.922579] x21: 0000000000000000 x20: ffffff85a1dafd00
[21295.922579] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[21295.922580] x17: 0000000000000001 x16: ffffffc010edad60
[21295.922580] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21295.922581] x13: 000039ce00003cc0 x12: ffffffff16262400
[21295.922581] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21295.922582] x9 : 0000000000000000 x8 : 0000000000000300
[21295.922582] x7 : 0000000000000000 x6 : 000000000000003f
[21295.922583] x5 : 0000000000000040 x4 : 0000000000000000
[21295.922583] x3 : 000000007cd31279 x2 : 0000000000000000
[21295.922584] x1 : ffffff85a1daf300 x0 : ffffff85a1daf300
[21295.922585] Call trace:
[21295.922585] skb_split (net/core/skbuff.c:3280)
[21379.068677] rcu: INFO: rcu_sched self-detected stall on CPU
[21379.071036] rcu:      1-....: (285172 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=129306
[21379.072911]   (t=285020 jiffies g=266325 q=3248)
[21379.072912] Task dump for CPU 1:
[21379.072913] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21379.072915] Call trace:
[21379.072931] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21379.072934] sched_show_task (kernel/sched/core.c:6101)
[21379.072937] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21379.072939] __memset (arch/arm64/lib/memset.S:153)
[21487.916072] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21487.918185] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21487.918213] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21487.918214] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21487.918215] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21487.918219] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:907)
[21487.918220] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[21487.918220] sp : ffffff85cf35f980
[21487.918221] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21487.918221] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21487.918222] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21487.918222] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[21487.918223] x21: ffffff85d31ed400 x20: ffffff85cf36ef00
[21487.918223] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[21487.918224] x17: 0000000000000001 x16: ffffffc010edad60
[21487.918224] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21487.918225] x13: 000039ce00003cc0 x12: ffffffff16262400
[21487.918225] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21487.918226] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[21487.918227] x7 : 00000000000000a0 x6 : 00000003bed4acf8
[21487.918227] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[21487.918228] x3 : 000000007cd21279 x2 : 0000000000000018
[21487.918228] x1 : 000000000000004e x0 : ffffff85d1b5f080
[21487.918229] Call trace:
[21487.918230] mptcp_data_ready (net/mptcp/mptcp_input.c:907)
[21559.079678] rcu: INFO: rcu_sched self-detected stall on CPU
[21559.081909] rcu:      1-....: (330175 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=150025
[21559.083604]   (t=330022 jiffies g=266325 q=3504)
[21559.083605] Task dump for CPU 1:
[21559.083606] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21559.083608] Call trace:
[21559.083612] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21559.083615] sched_show_task (kernel/sched/core.c:6101)
[21559.083617] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21559.083618] skb_split (net/core/skbuff.c:3239 (discriminator 1))
[21663.915098] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21663.929143] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21663.929171] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21663.929172] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21663.929172] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21663.929176] pc : skb_release_data (include/linux/skbuff.h:1388)
[21663.929177] lr : __kfree_skb (net/core/skbuff.c:630)
[21663.929177] sp : ffffff85cf35f940
[21663.929177] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21663.929178] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[21663.929178] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[21663.929179] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21663.929179] x21: 000000000383f80f x20: ffffff85ccc62e00
[21663.929180] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[21663.929180] x17: 0000000000000001 x16: ffffffc010edad60
[21663.929181] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21663.929181] x13: 000039ce00003cc0 x12: ffffffff16262400
[21663.929182] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21663.929182] x9 : 000000000383f80f x8 : 000000000000004e
[21663.929183] x7 : 00000000000000a0 x6 : 0000000000000008
[21663.929183] x5 : ffffff85a1daf300 x4 : 0000000000009a34
[21663.929184] x3 : 00000000001fffff x2 : ffffffc0119d7238
[21663.929184] x1 : 00000000000000c0 x0 : 0000000000000000
[21663.929185] Call trace:
[21663.929185] skb_release_data (include/linux/skbuff.h:1388)
[21739.090679] rcu: INFO: rcu_sched self-detected stall on CPU
[21739.106352] rcu:      1-....: (375175 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=171058
[21739.107298]   (t=375029 jiffies g=266325 q=3956)
[21739.107298] Task dump for CPU 1:
[21739.107299] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21739.107301] Call trace:
[21739.107304] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21739.107307] sched_show_task (kernel/sched/core.c:6101)
[21739.107309] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21739.107311] mptcp_data_ready (include/net/sock.h:1423)
[21839.914120] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21839.930519] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21839.930570] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21839.930570] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21839.930571] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21839.930574] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:1014)
[21839.930575] lr : mptcp_data_ready (include/linux/skbuff.h:1988)
[21839.930576] sp : ffffff85cf35f980
[21839.930576] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21839.930577] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21839.930577] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21839.930578] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21839.930578] x21: 000000000383f80f x20: ffffff85d31ed400
[21839.930579] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[21839.930579] x17: 0000000000000001 x16: ffffffc010edad60
[21839.930579] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21839.930580] x13: 000039ce00003cc0 x12: ffffffff16262400
[21839.930580] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21839.930581] x9 : 000000000383f80f x8 : 000000000000004e
[21839.930582] x7 : 00000000000000a0 x6 : 0000000000000008
[21839.930582] x5 : ffffff85a1dafd00 x4 : fe0fcb6f0383f80f
[21839.930583] x3 : ffffff85d6719c90 x2 : ffffff85a1daf328
[21839.930583] x1 : 000000007cd21279 x0 : ffffff85a1daf300
[21839.930584] Call trace:
[21839.930585] mptcp_data_ready (net/mptcp/mptcp_input.c:1014)
[21919.101680] rcu: INFO: rcu_sched self-detected stall on CPU
[21919.110663] rcu:      1-....: (420171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=190960
[21919.111406]   (t=420030 jiffies g=266325 q=4335)
[21919.111407] Task dump for CPU 1:
[21919.111407] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21919.111409] Call trace:
[21919.111413] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21919.111415] sched_show_task (kernel/sched/core.c:6101)
[21919.111416] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21919.111419] kmem_cache_alloc (mm/slub.c:2823)
[22015.913142] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22015.924876] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22015.925026] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22015.925027] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22015.925028] pstate: 80400005 (Nzcv daif +PAN -UAO)
[22015.925031] pc : skb_split (net/core/skbuff.c:3244 (discriminator 2))
[22015.925032] lr : skb_split (net/core/skbuff.c:3286)
[22015.925032] sp : ffffff85cf35f940
[22015.925032] x29: ffffff85cf35fff0 x28: 0000000000000034
[22015.925033] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22015.925033] x25: ffffff85d6719c90 x24: 0000000000010000
[22015.925034] x23: 0000000000000000 x22: 0000000000000000
[22015.925035] x21: 0000000000000000 x20: ffffff85a1dafd00
[22015.925035] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[22015.925036] x17: 0000000000000001 x16: ffffffc010edad60
[22015.925036] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22015.925037] x13: 000039ce00003cc0 x12: ffffffff16262400
[22015.925038] x11: 00000000038382bf x10: 00000000000000c0
[22015.925038] x9 : 0000000000000001 x8 : 0000000000000008
[22015.925039] x7 : 0000000000000070 x6 : 0000000000000005
[22015.925039] x5 : 0000000000008390 x4 : 0000000000000004
[22015.925040] x3 : 00000bce00002d40 x2 : ffffffff15ec9c00
[22015.925040] x1 : ffffff85ccc62f00 x0 : 000000000000b0d0
[22015.925041] Call trace:
[22015.925042] skb_split (net/core/skbuff.c:3244 (discriminator 2))
[22099.112680] rcu: INFO: rcu_sched self-detected stall on CPU
[22099.124232] rcu:      1-....: (465171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=211501
[22099.129635]   (t=465034 jiffies g=266325 q=5253)
[22099.129636] Task dump for CPU 1:
[22099.129637] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22099.129803] Call trace:
[22099.129963] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22099.129968] sched_show_task (kernel/sched/core.c:6101)
[22099.129970] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22099.129974] kmem_cache_free (include/linux/compiler.h:268 (discriminator 5))
[22207.912076] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22207.913725] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22207.913754] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22207.913755] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22207.913756] pstate: 20400005 (nzCv daif +PAN -UAO)
[22207.913771] pc : skb_split (include/linux/skbuff.h:1388)
[22207.913771] lr : skb_split (net/core/skbuff.c:3286)
[22207.913772] sp : ffffff85cf35f940
[22207.913772] x29: ffffff85cf35fff0 x28: 0000000000000034
[22207.913773] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22207.913773] x25: ffffff85d6719c90 x24: 0000000000010000
[22207.913774] x23: 0000000000000000 x22: 0000000000000000
[22207.913774] x21: 0000000000000000 x20: ffffff85a1dafd00
[22207.913775] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[22207.913775] x17: 0000000000000001 x16: ffffffc010edad60
[22207.913775] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22207.913776] x13: 000039ce00003cc0 x12: ffffffff16262400
[22207.913776] x11: 00000000038382bf x10: 00000000000000c0
[22207.913777] x9 : 0000000000000001 x8 : 0000000000000008
[22207.913777] x7 : 0000000000000070 x6 : 0000000000000005
[22207.913778] x5 : 000000000000b0d0 x4 : 0000000000000005
[22207.913778] x3 : 0000000000000008 x2 : ffffffff15ec9c00
[22207.913779] x1 : ffffff85ccc63400 x0 : 000000000000b0d0
[22207.913780] Call trace:
[22207.913780] skb_split (include/linux/skbuff.h:1388)
[22279.123679] rcu: INFO: rcu_sched self-detected stall on CPU
[22279.124669] rcu:      1-....: (510171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=231776
[22279.125441]   (t=510034 jiffies g=266325 q=5520)
[22279.125442] Task dump for CPU 1:
[22279.125442] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22279.125444] Call trace:
[22279.125448] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22279.125450] sched_show_task (kernel/sched/core.c:6101)
[22279.125452] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22279.125455] __kmalloc_track_caller (arch/arm64/include/asm/atomic_lse.h:317)
[22383.911097] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22383.912262] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22383.912291] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22383.912292] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22383.912293] pstate: 80400005 (Nzcv daif +PAN -UAO)
[22383.912298] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:688)
[22383.912299] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[22383.912300] sp : ffffff85cf35f980
[22383.912300] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22383.912300] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22383.912301] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22383.912301] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[22383.912302] x21: ffffff85d31ed400 x20: ffffff85cf36ef00
[22383.912302] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[22383.912303] x17: 0000000000000001 x16: ffffffc010edad60
[22383.912303] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22383.912304] x13: 000039ce00003cc0 x12: ffffffff16262400
[22383.912304] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22383.912305] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[22383.912305] x7 : 00000000000000a0 x6 : 000000066e62dd58
[22383.912306] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[22383.912306] x3 : 000000007cd21279 x2 : 0000000000000000
[22383.912307] x1 : 0000000000010000 x0 : ffffff85d1b5f080
[22383.912308] Call trace:
[22383.912309] mptcp_data_ready (net/mptcp/mptcp_input.c:688)
[22459.134678] rcu: INFO: rcu_sched self-detected stall on CPU
[22459.135619] rcu:      1-....: (555174 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=251634
[22459.136475]   (t=555037 jiffies g=266325 q=6012)
[22459.136476] Task dump for CPU 1:
[22459.136477] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22459.136479] Call trace:
[22459.136483] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22459.136485] sched_show_task (kernel/sched/core.c:6101)
[22459.136487] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22459.136490] kmem_cache_alloc (mm/slub.c:2772)
[22559.910119] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22559.911203] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22559.911230] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22559.911230] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22559.911231] pstate: 20400005 (nzCv daif +PAN -UAO)
[22559.911235] pc : skb_split (net/core/skbuff.c:3247)
[22559.911235] lr : skb_split (net/core/skbuff.c:3286)
[22559.911236] sp : ffffff85cf35f940
[22559.911236] x29: ffffff85cf35fff0 x28: 0000000000000034
[22559.911237] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[22559.911237] x25: ffffff85d6719c90 x24: 0000000000010000
[22559.911238] x23: 0000000000000000 x22: 0000000000000000
[22559.911238] x21: 0000000000000000 x20: ffffff85a1daf300
[22559.911239] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[22559.911239] x17: 0000000000000001 x16: ffffffc010edad60
[22559.911240] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22559.911240] x13: 000039ce00003cc0 x12: ffffffff16262400
[22559.911241] x11: 00000000038382bf x10: 00000000000000c0
[22559.911241] x9 : 0000000000000001 x8 : 0000000000000008
[22559.911242] x7 : 0000000000000060 x6 : 0000000000000003
[22559.911242] x5 : 000000000000803a x4 : 0000000000000003
[22559.911243] x3 : ffffff85ccc62f20 x2 : 0000000000000356
[22559.911243] x1 : ffffff85ccc62ec0 x0 : 000000000000803a
[22559.911244] Call trace:
[22559.911245] skb_split (net/core/skbuff.c:3247)
[22639.145676] rcu: INFO: rcu_sched self-detected stall on CPU
[22639.146553] rcu:      1-....: (600177 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=272376
[22639.147309]   (t=600040 jiffies g=266325 q=6479)
[22639.147310] Task dump for CPU 1:
[22639.147311] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22639.147313] Call trace:
[22639.147316] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22639.147318] sched_show_task (kernel/sched/core.c:6101)
[22639.147321] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22639.147323] mptcp_data_ready (net/mptcp/mptcp_input.c:1019)
[22735.909139] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22735.910153] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22735.910189] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22735.910190] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22735.910191] pstate: 60400005 (nZCv daif +PAN -UAO)
[22735.910195] pc : kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[22735.910197] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[22735.910198] sp : ffffff85cf35f950
[22735.910198] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22735.910199] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22735.910199] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22735.910200] x23: ffffff85cf36ef00 x22: ffffffc0111ce85c
[22735.910200] x21: ffffffff16676bc0 x20: ffffff85a1daf300
[22735.910201] x19: ffffff85ff75b200 x18: ffffff85d31c7720
[22735.910201] x17: 0000000000000001 x16: ffffffc010edad60
[22735.910202] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22735.910202] x13: 000039ce00003cc0 x12: ffffffff16262400
[22735.910202] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22735.910203] x9 : 000000000383f80f x8 : 000000000000004e
[22735.910203] x7 : 00000000000000a0 x6 : 000000077c3109fe
[22735.910204] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[22735.910204] x3 : 000000077c3109ff x2 : ffffff85a1daf300
[22735.910205] x1 : 000000077c3109fe x0 : ffffff85a1daf100
[22735.910206] Call trace:
[22735.910207] kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[22819.156674] rcu: INFO: rcu_sched self-detected stall on CPU
[22819.157603] rcu:      1-....: (645180 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=292064
[22819.158443]   (t=645043 jiffies g=266325 q=6921)
[22819.158444] Task dump for CPU 1:
[22819.158444] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22819.158446] Call trace:
[22819.158450] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22819.158452] sched_show_task (kernel/sched/core.c:6101)
[22819.158455] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22819.158456] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[22927.908071] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[22927.911116] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22927.911143] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22927.911143] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22927.911144] pstate: 40400005 (nZcv daif +PAN -UAO)
[22927.911148] pc : kfree (mm/slub.c:3026)
[22927.911150] lr : __kfree_skb (net/core/skbuff.c:630)
[22927.911150] sp : ffffff85cf35f950
[22927.911150] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22927.911151] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22927.911151] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22927.911152] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[22927.911152] x21: ffffffff17131880 x20: ffffff85ccc63400
[22927.911153] x19: ffffff85f8b42c00 x18: ffffff85d31c7720
[22927.911153] x17: 0000000000000001 x16: ffffffc010edad60
[22927.911154] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22927.911154] x13: 000039ce00003cc0 x12: ffffffff16262400
[22927.911155] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22927.911155] x9 : 000000000383f80f x8 : 000000000000004e
[22927.911156] x7 : 00000000000000a0 x6 : 0000000000000008
[22927.911156] x5 : ffffff85a1dafd00 x4 : 0000000000009a34
[22927.911157] x3 : 00000000001fffff x2 : ffffff85cf35f950
[22927.911157] x1 : 000000080e5aeaae x0 : ffffffc01183bfa8
[22927.911158] Call trace:
[22927.911159] kfree (mm/slub.c:3026)
[22999.167676] rcu: INFO: rcu_sched self-detected stall on CPU
[22999.171318] rcu:      1-....: (690183 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=311736
[22999.173046]   (t=690047 jiffies g=266325 q=7179)
[22999.173047] Task dump for CPU 1:
[22999.173048] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22999.173050] Call trace:
[22999.173053] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22999.173056] sched_show_task (kernel/sched/core.c:6101)
[22999.173058] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22999.173061] kfree (arch/arm64/include/asm/atomic_lse.h:317)
[23103.907091] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[23103.920965] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[23103.920992] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[23103.920993] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[23103.920994] pstate: 40400005 (nZcv daif +PAN -UAO)
[23103.920998] pc : kfree (include/linux/page-flags.h:182)
[23103.921000] lr : __kfree_skb (net/core/skbuff.c:630)
[23103.921000] sp : ffffff85cf35f950
[23103.921001] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[23103.921001] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[23103.921002] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[23103.921002] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[23103.921003] x21: ffffffff17131880 x20: ffffff85ccc62e00
[23103.921003] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[23103.921004] x17: 0000000000000001 x16: ffffffc010edad60
[23103.921004] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[23103.921005] x13: 000039ce00003cc0 x12: ffffffff16262400
[23103.921006] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[23103.921006] x9 : 000000000383f80f x8 : 000000000000004e
[23103.921007] x7 : 00000000000000a0 x6 : 0000000000000008
[23103.921007] x5 : ffffff85a1daf300 x4 : 0000000000009a34
[23103.921008] x3 : 00000000001fffff x2 : 0000000000000000
[23103.921008] x1 : 0000000000000000 x0 : ffffffff17202000
[23103.921009] Call trace:
[23103.921011] kfree (include/linux/page-flags.h:182)

The code is at https://github.com/arter97/x86-kernel/tree/5.4 (commit 9bc70a46b024621d1dc2a931940b9ac01ce6cbcf) and the defconfig is at https://github.com/arter97/x86-kernel/blob/5.4/arch/arm64/configs/oracle_defconfig.

(Ignore the repo name, it's misleading lol)

Thanks.