search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
889 stars 335 forks source link

NULL pointer dereference in MPTCP v0.96 #461

Closed arter97 closed 2 years ago

arter97 commented 2 years ago

Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.

As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).

Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.

Thanks.

[379830.687736] BUG: kernel NULL pointer dereference, address: 0000000000000078
[379830.687740] #PF: supervisor read access in kernel mode
[379830.687741] #PF: error_code(0x0000) - not-present page
[379830.687742] PGD 0 P4D 0 
[379830.687744] Oops: 0000 [#1] SMP
[379830.687745] CPU: 1 PID: 7079 Comm: tokio-runtime-w Tainted: P           O      5.4.170+ #1
[379830.687747] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2401 07/12/2019
[379830.687750] RIP: 0010:tcp_v4_send_reset+0x1d7/0xa90
[379830.687752] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
[379830.687754] RSP: 0018:ffff8884e9553c30 EFLAGS: 00010246
[379830.687755] RAX: 0000000000000000 RBX: ffff8881cb04d3c0 RCX: 0000000000008331
[379830.687756] RDX: 0000000000000000 RSI: ffff8883e7288600 RDI: 0000000000000000
[379830.687757] RBP: ffff8884e9553d10 R08: 000000000001dd7c R09: 000000004a9c0000
[379830.687758] R10: ffff8884e9553d28 R11: ffff88813cbadf10 R12: 0000000000000000
[379830.687759] R13: ffff8882a467210e R14: 0000000000000000 R15: ffff8883e7288600
[379830.687761] FS:  00007f78e5777640(0000) GS:ffff88882ea40000(0000) knlGS:0000000000000000
[379830.687762] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[379830.687763] CR2: 0000000000000078 CR3: 00000008222ea005 CR4: 00000000003726e0
[379830.687764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[379830.687765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[379830.687766] Call Trace:
[379830.687770]  ? __x64_sys_epoll_ctl+0xf2/0x2e60
[379830.687773]  ? __mod_timer+0x106/0x900
[379830.687774]  ? mptcp_v4_do_rcv+0x14f/0x3e0
[379830.687776]  ? mptcp_v4_do_rcv+0x14f/0x3e0
[379830.687777]  ? mptcp_backlog_rcv+0xb9/0xe0
[379830.687778]  ? _raw_spin_lock_bh+0x20/0x30
[379830.687780]  ? release_sock+0x8b/0x160
[379830.687781]  ? mptcp_close+0x387/0xaf0
[379830.687783]  ? fsnotify_destroy_marks+0x1d/0xb40
[379830.687785]  ? inet_release+0x29/0x50
[379830.687787]  ? sock_close+0x39/0xa0
[379830.687789]  ? ____fput+0xb7/0x240
[379830.687790]  ? task_work_run+0x7c/0xa0
[379830.687792]  ? do_syscall_64+0x495/0x5a0
[379830.687793]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[379830.687794] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_set xt_geoip(O) xt_multiport ip_set_hash_ip ip_set nvme_fabrics xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink btrfs xor raid6_pq xfs intel_rapl_msr intel_rapl_common zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) x86_pkg_temp_thermal zcommon(PO) eeepc_wmi intel_powerclamp znvpair(PO) asus_wmi mei_hdcp sparse_keymap wmi_bmof mxm_wmi intel_wmi_thunderbolt coretemp spl(O) i40e mei_me kvm_intel xhci_pci mei xhci_hcd ahci i2c_i801 libahci wmi video mac_hid acpi_pad sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4
[379830.687816] CR2: 0000000000000078
[379830.687818] ---[ end trace 7172cb6f6f0ce936 ]---
[379830.687820] RIP: 0010:tcp_v4_send_reset+0x1d7/0xa90
[379830.687821] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
[379830.687823] RSP: 0018:ffff8884e9553c30 EFLAGS: 00010246
[379830.687824] RAX: 0000000000000000 RBX: ffff8881cb04d3c0 RCX: 0000000000008331
[379830.687825] RDX: 0000000000000000 RSI: ffff8883e7288600 RDI: 0000000000000000
[379830.687826] RBP: ffff8884e9553d10 R08: 000000000001dd7c R09: 000000004a9c0000
[379830.687827] R10: ffff8884e9553d28 R11: ffff88813cbadf10 R12: 0000000000000000
[379830.687828] R13: ffff8882a467210e R14: 0000000000000000 R15: ffff8883e7288600
[379830.687829] FS:  00007f78e5777640(0000) GS:ffff88882ea40000(0000) knlGS:0000000000000000
[379830.687830] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[379830.687831] CR2: 0000000000000078 CR3: 00000008222ea005 CR4: 00000000003726e0
[379830.687832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[379830.687833] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
matttbe commented 2 years ago

Hello,

Thank you for this bug report.

Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.

Do you mean you never saw this issue before and it could be due to the recent update from the v5.4.155 to v5.4.170?

As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).

OK so not easy to reproduce, a fun one :)

Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.

By chance, do you still have your build environment? If yes, then could you try to decode the stacktrace if you don't mind? You can use decode_stracktrace.sh for that, e.g.:

./scripts/decode_stacktrace.sh "${BUILD_DIR}/vmlinux" "${KERNEL_SRC}" "${KERNEL_SRC}"
arter97 commented 2 years ago

Hi :)

Hello,

Thank you for this bug report.

Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.

Do you mean you never saw this issue before and it could be due to the recent update from the v5.4.155 to v5.4.170?

I'm merging v5.4 from time to time on my own, and the last kernel was v5.4.161. However, since the issue appears after long uptime, and the fact that I had shadowsocks-rust upgraded at the same time (which was left untouched for about year), I don't think bisecting is a feasible solution here. Older kernels may have the same issue.

As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).

OK so not easy to reproduce, a fun one :)

Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.

By chance, do you still have your build environment? If yes, then could you try to decode the stacktrace if you don't mind? You can use decode_stracktrace.sh for that, e.g.:

Unfortunately, no :(

I'll be setting up a new kernel build with some debug options enabled so that we can find the culprit easier. I'll update you when another bug happens.

Thanks.

./scripts/decode_stacktrace.sh "${BUILD_DIR}/vmlinux" "${KERNEL_SRC}" "${KERNEL_SRC}"
arter97 commented 2 years ago

Hi.

I just had the same BUG, but due to my stupid mistake, it was running the kernel without debug configs enabled, apologies :(

The uptime this time was 430560. Funny thing is that it only happens on the server. The client (running shadowsocks-rust's sslocal) seems fine.

Anyways, I'll replace the kernel and dump the relevant info when it happens again.

arter97 commented 2 years ago

Well, this must have been an upstream issue.

The issue seems to be gone after merging v5.4.173. The uptime is now 1456272 seconds, much longer than what would have oops'ed before.

I'll close the issue (and re-open in case I was wrong and it happens again).

Btw, when I git log -- net/, following commits appear between the last kernel:

44065cc11797 net: udp: fix alignment problem in udp4_seq_show()
0ad45baead37 ip6_vti: initialize __ip6_tnl_parm struct in vti6_siocdevprivate
f0e57098243c ipv6: Do cleanup if attribute validation fails in multipath route
c94999cfbbbe ipv6: Continue processing multipath route even if gateway attribute is invalid
2a6a811a45fd phonet: refcount leak in pep_sock_accep
c0db2e1e60c6 sch_qfq: prevent shift-out-of-bounds in qfq_init_qdisc
bcbfc7780047 batman-adv: mcast: don't send link-local multicast to mcast routers
76936ddb4913 lwtunnel: Validate RTA_ENCAP_TYPE attribute length
2ebd777513d9 ipv6: Check attribute length for RTA_GATEWAY when deleting multipath route
a02d2be7eb48 ipv6: Check attribute length for RTA_GATEWAY in multipath route
34224e936a9d ipv4: Check attribute length for RTA_FLOW in multipath route
125d91f07233 ipv4: Check attribute length for RTA_GATEWAY in multipath route
38fbb1561d66 mac80211: initialize variable have_higher_than_11mbit
matttbe commented 2 years ago

Thank you for this last comment!

mptcp_v0.96 is now on top of v5.4.178.

arter97 commented 2 years ago

Welp, seems like it's not fixed after all.

Weird that the call trace still prints '?' when I enabled frame pointers....

[142248.500615] BUG: kernel NULL pointer dereference, address: 0000000000000078
[142248.500619] #PF: supervisor read access in kernel mode
[142248.500620] #PF: error_code(0x0000) - not-present page
[142248.500621] PGD 0 P4D 0 
[142248.500623] Oops: 0000 [#1] SMP
[142248.500624] CPU: 2 PID: 3166 Comm: tokio-runtime-w Tainted: P           O      5.4.181+ #1
[142248.500625] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2401 07/12/2019
[142248.500629] RIP: 0010:tcp_v4_send_reset+0x1d7/0xa90
[142248.500630] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
[142248.500632] RSP: 0018:ffff8884ea47fa80 EFLAGS: 00010246
[142248.500634] RAX: 0000000000000000 RBX: ffff8882b9c06640 RCX: 000000000000d54c
[142248.500634] RDX: 0000000000000000 RSI: ffff88844c2f3f00 RDI: 0000000000000000
[142248.500635] RBP: ffff8884ea47fb60 R08: 00000000fe99e14a R09: 000000004a9c0000
[142248.500636] R10: ffff8884ea47fb80 R11: ffff8884ee63a440 R12: ffff88844c2f3f00
[142248.500637] R13: ffff88800928210e R14: 0000000000000000 R15: ffff88844c2f3f00
[142248.500639] FS:  00007fbf72d10640(0000) GS:ffff88882ea80000(0000) knlGS:0000000000000000
[142248.500640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[142248.500641] CR2: 0000000000000078 CR3: 000000080196f001 CR4: 00000000003726e0
[142248.500642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[142248.500642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[142248.500643] Call Trace:
[142248.500647]  ? ip_finish_output.part.0+0x2890/0x2890
[142248.500649]  ? nf_hook_slow+0x3b/0xb0
[142248.500651]  ? __ip_queue_xmit+0x221/0x550
[142248.500653]  ? select_task_rq_fair+0x90c/0xc80
[142248.500655]  mptcp_v4_do_rcv+0x151/0x3e0
[142248.500656]  ? mptcp_v4_do_rcv+0x151/0x3e0
[142248.500657]  ? ip_queue_xmit+0x10/0x20
[142248.500659]  tcp_v4_do_rcv+0x3a5/0xca0
[142248.500661]  ? __x32_compat_sys_fanotify_mark+0xad0/0xad0
[142248.500662]  mptcp_backlog_rcv+0xbe/0xe0
[142248.500663]  ? mptcp_backlog_rcv+0xbe/0xe0
[142248.500664]  ? tcp_close_state+0xa0/0x280
[142248.500666]  release_sock+0x96/0x180
[142248.500667]  mptcp_close+0x39d/0xb20
[142248.500669]  ? inode_wait_for_writeback+0x40/0xd0
[142248.500670]  tcp_close+0x625/0x10d0
[142248.500672]  ? kmem_cache_free+0x1f4/0x250
[142248.500673]  inet_release+0x30/0x60
[142248.500674]  sock_close+0x3f/0xb0
[142248.500676]  ____fput+0xc2/0x250
[142248.500678]  task_work_run+0x8f/0xb0
[142248.500680]  do_syscall_64+0x4a1/0x5c0
[142248.500682]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[142248.500684] RIP: 0033:0x7fbf7363213b
[142248.500685] Code: 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 43 b9 f7 ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 91 b9 f7 ff 8b 44
[142248.500686] RSP: 002b:00007fbf72d0c390 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[142248.500687] RAX: 0000000000000000 RBX: 000055e4a1215750 RCX: 00007fbf7363213b
[142248.500688] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000000000009f
[142248.500689] RBP: 00007fbf72d0f770 R08: 0000000000000000 R09: 0000000000000000
[142248.500690] R10: 0000000000000000 R11: 0000000000000293 R12: 000055e4a1215c90
[142248.500691] R13: 0000000000000568 R14: 000055e4a113e980 R15: 000000000000009f
[142248.500692] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_set xt_geoip(O) xt_multiport ip_set_hash_ip ip_set nvme_fabrics xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink zfs(PO) zunicode(PO) intel_rapl_msr zzstd(O) zlua(O) zavl(PO) intel_rapl_common icp(PO) btrfs x86_pkg_temp_thermal eeepc_wmi intel_powerclamp asus_wmi xor zcommon(PO) xfs raid6_pq coretemp znvpair(PO) sparse_keymap wmi_bmof intel_wmi_thunderbolt mxm_wmi spl(O) kvm_intel xhci_pci i40e mei_me i2c_i801 xhci_hcd ahci mei libahci wmi video acpi_pad mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4
[142248.500711] CR2: 0000000000000078
[142248.500712] ---[ end trace 525f8b090c66add0 ]---
matttbe commented 2 years ago

Do you have the latest fixes from mptcp_v0.96 branch?

May you run decode_stracktrace.sh script if you don't mind? See my first reply here for more details about it.

matttbe commented 2 years ago

Maybe fixed with e5755ac577dcc6685bfeb5e6d454db98bd7a7727

arter97 commented 2 years ago 
[142248.500615] BUG: kernel NULL pointer dereference, address: 0000000000000078
[142248.500619] #PF: supervisor read access in kernel mode
[142248.500620] #PF: error_code(0x0000) - not-present page
[142248.500621] PGD 0 P4D 0
[142248.500623] Oops: 0000 [#1] SMP
[142248.500624] CPU: 2 PID: 3166 Comm: tokio-runtime-w Tainted: P           O      5.4.181+ #1
[142248.500625] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2401 07/12/2019
[142248.500629] RIP: 0010:tcp_v4_send_reset (/home/arter97/kernel/liquorix/net/ipv4/tcp_ipv4.c:690) 
[142248.500630] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
All code
========
   0:   83 c4 20                add    $0x20,%esp
   3:   48 85 c0                test   %rax,%rax
   6:   74 05                   je     0xd
   8:   0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)
   d:   48 8d 65 d0             lea    -0x30(%rbp),%rsp
  11:   5b                      pop    %rbx
  12:   41 5a                   pop    %r10
  14:   41 5c                   pop    %r12
  16:   41 5d                   pop    %r13
  18:   41 5e                   pop    %r14
  1a:   41 5f                   pop    %r15
  1c:   5d                      pop    %rbp
  1d:   49 8d 62 f8             lea    -0x8(%r10),%rsp
  21:   c3                      ret    
  22:   48 8b 46 78             mov    0x78(%rsi),%rax
  26:   48 83 e0 fe             and    $0xfffffffffffffffe,%rax
  2a:*  66 83 78 78 02          cmpw   $0x2,0x78(%rax)      <-- trapping instruction
  2f:   75 dc                   jne    0xd
  31:   e9 69 fe ff ff          jmp    0xfffffffffffffe9f
  36:   41 8b 45 08             mov    0x8(%r13),%eax
  3a:   89 45 ac                mov    %eax,-0x54(%rbp)
  3d:   e9                      .byte 0xe9
  3e:   eb fe                   jmp    0x3e

Code starting with the faulting instruction
===========================================
   0:   66 83 78 78 02          cmpw   $0x2,0x78(%rax)
   5:   75 dc                   jne    0xffffffffffffffe3
   7:   e9 69 fe ff ff          jmp    0xfffffffffffffe75
   c:   41 8b 45 08             mov    0x8(%r13),%eax
  10:   89 45 ac                mov    %eax,-0x54(%rbp)
  13:   e9                      .byte 0xe9
  14:   eb fe                   jmp    0x14
[142248.500632] RSP: 0018:ffff8884ea47fa80 EFLAGS: 00010246
[142248.500634] RAX: 0000000000000000 RBX: ffff8882b9c06640 RCX: 000000000000d54c
[142248.500634] RDX: 0000000000000000 RSI: ffff88844c2f3f00 RDI: 0000000000000000
[142248.500635] RBP: ffff8884ea47fb60 R08: 00000000fe99e14a R09: 000000004a9c0000
[142248.500636] R10: ffff8884ea47fb80 R11: ffff8884ee63a440 R12: ffff88844c2f3f00
[142248.500637] R13: ffff88800928210e R14: 0000000000000000 R15: ffff88844c2f3f00
[142248.500639] FS:  00007fbf72d10640(0000) GS:ffff88882ea80000(0000) knlGS:0000000000000000
[142248.500640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[142248.500641] CR2: 0000000000000078 CR3: 000000080196f001 CR4: 00000000003726e0
[142248.500642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[142248.500642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[142248.500643] Call Trace:
[142248.500647] ? ip_finish_output.part.0 (/home/arter97/kernel/liquorix/net/ipv4/ip_output.c:320) 
[142248.500649] ? nf_hook_slow (/home/arter97/kernel/liquorix/include/linux/netfilter.h:135) 
[142248.500651] ? __ip_queue_xmit (/home/arter97/kernel/liquorix/net/ipv4/ip_output.c:550) 
[142248.500653] ? select_task_rq_fair (/home/arter97/kernel/liquorix/kernel/sched/fair.c:6583) 
[142248.500655] mptcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ipv4.c:248) 
[142248.500656] ? mptcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ipv4.c:248) 
[142248.500657] ? ip_queue_xmit (/home/arter97/kernel/liquorix/include/net/ip.h:239) 
[142248.500659] tcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/ipv4/tcp_ipv4.c:1603) 
[142248.500661] ? __x32_compat_sys_fanotify_mark (/home/arter97/kernel/liquorix/fs/eventpoll.c:763) 
[142248.500662] mptcp_backlog_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ctrl.c:1156) 
[142248.500663] ? mptcp_backlog_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ctrl.c:1156) 
[142248.500664] ? tcp_close_state (/home/arter97/kernel/liquorix/net/ipv4/tcp.c:2396) 
[142248.500666] release_sock (/home/arter97/kernel/liquorix/net/core/sock.c:2493) 
[142248.500667] mptcp_close (/home/arter97/kernel/liquorix/arch/x86/include/asm/preempt.h:79) 
[142248.500669] ? inode_wait_for_writeback (/home/arter97/kernel/liquorix/fs/fs-writeback.c:1341) 
[142248.500670] tcp_close (/home/arter97/kernel/liquorix/net/ipv4/tcp.c:2451) 
[142248.500672] ? kmem_cache_free (/home/arter97/kernel/liquorix/mm/slub.c:3047) 
[142248.500673] inet_release (/home/arter97/kernel/liquorix/net/ipv4/af_inet.c:433) 
[142248.500674] sock_close (/home/arter97/kernel/liquorix/net/socket.c:593) 
[142248.500676] ____fput (/home/arter97/kernel/liquorix/fs/file_table.c:282) 
[142248.500678] task_work_run (/home/arter97/kernel/liquorix/kernel/task_work.c:115 (discriminator 1)) 
[142248.500680] do_syscall_64 (/home/arter97/kernel/liquorix/include/linux/tracehook.h:188) 
[142248.500682] entry_SYSCALL_64_after_hwframe (/home/arter97/kernel/liquorix/arch/x86/entry/entry_64.S:185) 
[142248.500684] RIP: 0033:0x7fbf7363213b
[142248.500685] Code: 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 43 b9 f7 ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 91 b9 f7 ff 8b 44
All code
========
   0:   03 00                   add    (%rax),%eax
   2:   00 00                   add    %al,(%rax)
   4:   0f 05                   syscall 
   6:   48 3d 00 f0 ff ff       cmp    $0xfffffffffffff000,%rax
   c:   77 41                   ja     0x4f
   e:   c3                      ret    
   f:   48 83 ec 18             sub    $0x18,%rsp
  13:   89 7c 24 0c             mov    %edi,0xc(%rsp)
  17:   e8 43 b9 f7 ff          call   0xfffffffffff7b95f
  1c:   8b 7c 24 0c             mov    0xc(%rsp),%edi
  20:   41 89 c0                mov    %eax,%r8d
  23:   b8 03 00 00 00          mov    $0x3,%eax
  28:   0f 05                   syscall 
  2a:*  48 3d 00 f0 ff ff       cmp    $0xfffffffffffff000,%rax     <-- trapping instruction
  30:   77 35                   ja     0x67
  32:   44 89 c7                mov    %r8d,%edi
  35:   89 44 24 0c             mov    %eax,0xc(%rsp)
  39:   e8 91 b9 f7 ff          call   0xfffffffffff7b9cf
  3e:   8b                      .byte 0x8b
  3f:   44                      rex.R

Code starting with the faulting instruction
===========================================
   0:   48 3d 00 f0 ff ff       cmp    $0xfffffffffffff000,%rax
   6:   77 35                   ja     0x3d
   8:   44 89 c7                mov    %r8d,%edi
   b:   89 44 24 0c             mov    %eax,0xc(%rsp)
   f:   e8 91 b9 f7 ff          call   0xfffffffffff7b9a5
  14:   8b                      .byte 0x8b
  15:   44                      rex.R
[142248.500686] RSP: 002b:00007fbf72d0c390 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[142248.500687] RAX: 0000000000000000 RBX: 000055e4a1215750 RCX: 00007fbf7363213b
[142248.500688] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000000000009f
[142248.500689] RBP: 00007fbf72d0f770 R08: 0000000000000000 R09: 0000000000000000
[142248.500690] R10: 0000000000000000 R11: 0000000000000293 R12: 000055e4a1215c90
[142248.500691] R13: 0000000000000568 R14: 000055e4a113e980 R15: 000000000000009f
[142248.500692] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_set xt_geoip(O) xt_multiport ip_set_hash_ip ip_set nvme_fabrics xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink zfs(PO) zunicode(PO) intel_rapl_msr zzstd(O) zlua(O) zavl(PO) intel_rapl_common icp(PO) btrfs x86_pkg_temp_thermal eeepc_wmi intel_powerclamp asus_wmi xor zcommon(PO) xfs raid6_pq coretemp znvpair(PO) sparse_keymap wmi_bmof intel_wmi_thunderbolt mxm_wmi spl(O) kvm_intel xhci_pci i40e mei_me i2c_i801 xhci_hcd ahci mei libahci wmi video acpi_pad mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4
[142248.500711] CR2: 0000000000000078
[142248.500712] ---[ end trace 525f8b090c66add0 ]---

This is what I get. Unfortunately, by another dumb mistake, I had rm -rf uncommented on my build script so the obj files were missing. I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.

I'll build a new kernel with the latest trunk merged. Thanks.

matttbe commented 2 years ago

I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.

It seems this is working fine :-)

Thank you for the output! It really looks like it is linked to e5755ac577dcc6685bfeb5e6d454db98bd7a7727. Do you mind if I close the ticket now and you re-open it if you still have the issue?

arter97 commented 2 years ago

I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.

It seems this is working fine :-)

Thank you for the output! It really looks like it is linked to e5755ac. Do you mind if I close the ticket now and you re-open it if you still have the issue?

Not at all.

I'm quite surprised that this is triggered so late down the road. I've been using v5.4 with MPTCP for years at this point :)

Anyways, I'm using the latest kernel and everything seems fine now. I'll try to use it without reboots for weeks and report back if anything happens.

Thanks!

matttbe commented 2 years ago

The bug happens in very specific conditions, that's maybe why :)

arter97 commented 2 years ago

Uptime is 6 weeks now, no issues so far. Must be fixed :)