Closed arter97 closed 2 years ago
Hello,
Thank you for this bug report.
Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.
Do you mean you never saw this issue before and it could be due to the recent update from the v5.4.155 to v5.4.170?
As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).
OK so not easy to reproduce, a fun one :)
Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.
By chance, do you still have your build environment?
If yes, then could you try to decode the stacktrace if you don't mind? You can use decode_stracktrace.sh
for that, e.g.:
./scripts/decode_stacktrace.sh "${BUILD_DIR}/vmlinux" "${KERNEL_SRC}" "${KERNEL_SRC}"
Hi :)
Hello,
Thank you for this bug report.
Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.
Do you mean you never saw this issue before and it could be due to the recent update from the v5.4.155 to v5.4.170?
I'm merging v5.4 from time to time on my own, and the last kernel was v5.4.161. However, since the issue appears after long uptime, and the fact that I had shadowsocks-rust upgraded at the same time (which was left untouched for about year), I don't think bisecting is a feasible solution here. Older kernels may have the same issue.
As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).
OK so not easy to reproduce, a fun one :)
Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.
By chance, do you still have your build environment? If yes, then could you try to decode the stacktrace if you don't mind? You can use
decode_stracktrace.sh
for that, e.g.:
Unfortunately, no :(
I'll be setting up a new kernel build with some debug options enabled so that we can find the culprit easier. I'll update you when another bug happens.
Thanks.
./scripts/decode_stacktrace.sh "${BUILD_DIR}/vmlinux" "${KERNEL_SRC}" "${KERNEL_SRC}"
Hi.
I just had the same BUG, but due to my stupid mistake, it was running the kernel without debug configs enabled, apologies :(
The uptime this time was 430560. Funny thing is that it only happens on the server. The client (running shadowsocks-rust's sslocal) seems fine.
Anyways, I'll replace the kernel and dump the relevant info when it happens again.
Well, this must have been an upstream issue.
The issue seems to be gone after merging v5.4.173. The uptime is now 1456272 seconds, much longer than what would have oops'ed before.
I'll close the issue (and re-open in case I was wrong and it happens again).
Btw, when I git log -- net/, following commits appear between the last kernel:
44065cc11797 net: udp: fix alignment problem in udp4_seq_show()
0ad45baead37 ip6_vti: initialize __ip6_tnl_parm struct in vti6_siocdevprivate
f0e57098243c ipv6: Do cleanup if attribute validation fails in multipath route
c94999cfbbbe ipv6: Continue processing multipath route even if gateway attribute is invalid
2a6a811a45fd phonet: refcount leak in pep_sock_accep
c0db2e1e60c6 sch_qfq: prevent shift-out-of-bounds in qfq_init_qdisc
bcbfc7780047 batman-adv: mcast: don't send link-local multicast to mcast routers
76936ddb4913 lwtunnel: Validate RTA_ENCAP_TYPE attribute length
2ebd777513d9 ipv6: Check attribute length for RTA_GATEWAY when deleting multipath route
a02d2be7eb48 ipv6: Check attribute length for RTA_GATEWAY in multipath route
34224e936a9d ipv4: Check attribute length for RTA_FLOW in multipath route
125d91f07233 ipv4: Check attribute length for RTA_GATEWAY in multipath route
38fbb1561d66 mac80211: initialize variable have_higher_than_11mbit
Thank you for this last comment!
mptcp_v0.96
is now on top of v5.4.178.
Welp, seems like it's not fixed after all.
Weird that the call trace still prints '?' when I enabled frame pointers....
[142248.500615] BUG: kernel NULL pointer dereference, address: 0000000000000078
[142248.500619] #PF: supervisor read access in kernel mode
[142248.500620] #PF: error_code(0x0000) - not-present page
[142248.500621] PGD 0 P4D 0
[142248.500623] Oops: 0000 [#1] SMP
[142248.500624] CPU: 2 PID: 3166 Comm: tokio-runtime-w Tainted: P O 5.4.181+ #1
[142248.500625] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2401 07/12/2019
[142248.500629] RIP: 0010:tcp_v4_send_reset+0x1d7/0xa90
[142248.500630] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
[142248.500632] RSP: 0018:ffff8884ea47fa80 EFLAGS: 00010246
[142248.500634] RAX: 0000000000000000 RBX: ffff8882b9c06640 RCX: 000000000000d54c
[142248.500634] RDX: 0000000000000000 RSI: ffff88844c2f3f00 RDI: 0000000000000000
[142248.500635] RBP: ffff8884ea47fb60 R08: 00000000fe99e14a R09: 000000004a9c0000
[142248.500636] R10: ffff8884ea47fb80 R11: ffff8884ee63a440 R12: ffff88844c2f3f00
[142248.500637] R13: ffff88800928210e R14: 0000000000000000 R15: ffff88844c2f3f00
[142248.500639] FS: 00007fbf72d10640(0000) GS:ffff88882ea80000(0000) knlGS:0000000000000000
[142248.500640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[142248.500641] CR2: 0000000000000078 CR3: 000000080196f001 CR4: 00000000003726e0
[142248.500642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[142248.500642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[142248.500643] Call Trace:
[142248.500647] ? ip_finish_output.part.0+0x2890/0x2890
[142248.500649] ? nf_hook_slow+0x3b/0xb0
[142248.500651] ? __ip_queue_xmit+0x221/0x550
[142248.500653] ? select_task_rq_fair+0x90c/0xc80
[142248.500655] mptcp_v4_do_rcv+0x151/0x3e0
[142248.500656] ? mptcp_v4_do_rcv+0x151/0x3e0
[142248.500657] ? ip_queue_xmit+0x10/0x20
[142248.500659] tcp_v4_do_rcv+0x3a5/0xca0
[142248.500661] ? __x32_compat_sys_fanotify_mark+0xad0/0xad0
[142248.500662] mptcp_backlog_rcv+0xbe/0xe0
[142248.500663] ? mptcp_backlog_rcv+0xbe/0xe0
[142248.500664] ? tcp_close_state+0xa0/0x280
[142248.500666] release_sock+0x96/0x180
[142248.500667] mptcp_close+0x39d/0xb20
[142248.500669] ? inode_wait_for_writeback+0x40/0xd0
[142248.500670] tcp_close+0x625/0x10d0
[142248.500672] ? kmem_cache_free+0x1f4/0x250
[142248.500673] inet_release+0x30/0x60
[142248.500674] sock_close+0x3f/0xb0
[142248.500676] ____fput+0xc2/0x250
[142248.500678] task_work_run+0x8f/0xb0
[142248.500680] do_syscall_64+0x4a1/0x5c0
[142248.500682] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[142248.500684] RIP: 0033:0x7fbf7363213b
[142248.500685] Code: 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 43 b9 f7 ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 91 b9 f7 ff 8b 44
[142248.500686] RSP: 002b:00007fbf72d0c390 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[142248.500687] RAX: 0000000000000000 RBX: 000055e4a1215750 RCX: 00007fbf7363213b
[142248.500688] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000000000009f
[142248.500689] RBP: 00007fbf72d0f770 R08: 0000000000000000 R09: 0000000000000000
[142248.500690] R10: 0000000000000000 R11: 0000000000000293 R12: 000055e4a1215c90
[142248.500691] R13: 0000000000000568 R14: 000055e4a113e980 R15: 000000000000009f
[142248.500692] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_set xt_geoip(O) xt_multiport ip_set_hash_ip ip_set nvme_fabrics xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink zfs(PO) zunicode(PO) intel_rapl_msr zzstd(O) zlua(O) zavl(PO) intel_rapl_common icp(PO) btrfs x86_pkg_temp_thermal eeepc_wmi intel_powerclamp asus_wmi xor zcommon(PO) xfs raid6_pq coretemp znvpair(PO) sparse_keymap wmi_bmof intel_wmi_thunderbolt mxm_wmi spl(O) kvm_intel xhci_pci i40e mei_me i2c_i801 xhci_hcd ahci mei libahci wmi video acpi_pad mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4
[142248.500711] CR2: 0000000000000078
[142248.500712] ---[ end trace 525f8b090c66add0 ]---
Do you have the latest fixes from mptcp_v0.96
branch?
May you run decode_stracktrace.sh
script if you don't mind? See my first reply here for more details about it.
Maybe fixed with e5755ac577dcc6685bfeb5e6d454db98bd7a7727
[142248.500615] BUG: kernel NULL pointer dereference, address: 0000000000000078
[142248.500619] #PF: supervisor read access in kernel mode
[142248.500620] #PF: error_code(0x0000) - not-present page
[142248.500621] PGD 0 P4D 0
[142248.500623] Oops: 0000 [#1] SMP
[142248.500624] CPU: 2 PID: 3166 Comm: tokio-runtime-w Tainted: P O 5.4.181+ #1
[142248.500625] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2401 07/12/2019
[142248.500629] RIP: 0010:tcp_v4_send_reset (/home/arter97/kernel/liquorix/net/ipv4/tcp_ipv4.c:690)
[142248.500630] Code: 83 c4 20 48 85 c0 74 05 0f 1f 44 00 00 48 8d 65 d0 5b 41 5a 41 5c 41 5d 41 5e 41 5f 5d 49 8d 62 f8 c3 48 8b 46 78 48 83 e0 fe <66> 83 78 78 02 75 dc e9 69 fe ff ff 41 8b 45 08 89 45 ac e9 eb fe
All code
========
0: 83 c4 20 add $0x20,%esp
3: 48 85 c0 test %rax,%rax
6: 74 05 je 0xd
8: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
d: 48 8d 65 d0 lea -0x30(%rbp),%rsp
11: 5b pop %rbx
12: 41 5a pop %r10
14: 41 5c pop %r12
16: 41 5d pop %r13
18: 41 5e pop %r14
1a: 41 5f pop %r15
1c: 5d pop %rbp
1d: 49 8d 62 f8 lea -0x8(%r10),%rsp
21: c3 ret
22: 48 8b 46 78 mov 0x78(%rsi),%rax
26: 48 83 e0 fe and $0xfffffffffffffffe,%rax
2a:* 66 83 78 78 02 cmpw $0x2,0x78(%rax) <-- trapping instruction
2f: 75 dc jne 0xd
31: e9 69 fe ff ff jmp 0xfffffffffffffe9f
36: 41 8b 45 08 mov 0x8(%r13),%eax
3a: 89 45 ac mov %eax,-0x54(%rbp)
3d: e9 .byte 0xe9
3e: eb fe jmp 0x3e
Code starting with the faulting instruction
===========================================
0: 66 83 78 78 02 cmpw $0x2,0x78(%rax)
5: 75 dc jne 0xffffffffffffffe3
7: e9 69 fe ff ff jmp 0xfffffffffffffe75
c: 41 8b 45 08 mov 0x8(%r13),%eax
10: 89 45 ac mov %eax,-0x54(%rbp)
13: e9 .byte 0xe9
14: eb fe jmp 0x14
[142248.500632] RSP: 0018:ffff8884ea47fa80 EFLAGS: 00010246
[142248.500634] RAX: 0000000000000000 RBX: ffff8882b9c06640 RCX: 000000000000d54c
[142248.500634] RDX: 0000000000000000 RSI: ffff88844c2f3f00 RDI: 0000000000000000
[142248.500635] RBP: ffff8884ea47fb60 R08: 00000000fe99e14a R09: 000000004a9c0000
[142248.500636] R10: ffff8884ea47fb80 R11: ffff8884ee63a440 R12: ffff88844c2f3f00
[142248.500637] R13: ffff88800928210e R14: 0000000000000000 R15: ffff88844c2f3f00
[142248.500639] FS: 00007fbf72d10640(0000) GS:ffff88882ea80000(0000) knlGS:0000000000000000
[142248.500640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[142248.500641] CR2: 0000000000000078 CR3: 000000080196f001 CR4: 00000000003726e0
[142248.500642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[142248.500642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[142248.500643] Call Trace:
[142248.500647] ? ip_finish_output.part.0 (/home/arter97/kernel/liquorix/net/ipv4/ip_output.c:320)
[142248.500649] ? nf_hook_slow (/home/arter97/kernel/liquorix/include/linux/netfilter.h:135)
[142248.500651] ? __ip_queue_xmit (/home/arter97/kernel/liquorix/net/ipv4/ip_output.c:550)
[142248.500653] ? select_task_rq_fair (/home/arter97/kernel/liquorix/kernel/sched/fair.c:6583)
[142248.500655] mptcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ipv4.c:248)
[142248.500656] ? mptcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ipv4.c:248)
[142248.500657] ? ip_queue_xmit (/home/arter97/kernel/liquorix/include/net/ip.h:239)
[142248.500659] tcp_v4_do_rcv (/home/arter97/kernel/liquorix/net/ipv4/tcp_ipv4.c:1603)
[142248.500661] ? __x32_compat_sys_fanotify_mark (/home/arter97/kernel/liquorix/fs/eventpoll.c:763)
[142248.500662] mptcp_backlog_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ctrl.c:1156)
[142248.500663] ? mptcp_backlog_rcv (/home/arter97/kernel/liquorix/net/mptcp/mptcp_ctrl.c:1156)
[142248.500664] ? tcp_close_state (/home/arter97/kernel/liquorix/net/ipv4/tcp.c:2396)
[142248.500666] release_sock (/home/arter97/kernel/liquorix/net/core/sock.c:2493)
[142248.500667] mptcp_close (/home/arter97/kernel/liquorix/arch/x86/include/asm/preempt.h:79)
[142248.500669] ? inode_wait_for_writeback (/home/arter97/kernel/liquorix/fs/fs-writeback.c:1341)
[142248.500670] tcp_close (/home/arter97/kernel/liquorix/net/ipv4/tcp.c:2451)
[142248.500672] ? kmem_cache_free (/home/arter97/kernel/liquorix/mm/slub.c:3047)
[142248.500673] inet_release (/home/arter97/kernel/liquorix/net/ipv4/af_inet.c:433)
[142248.500674] sock_close (/home/arter97/kernel/liquorix/net/socket.c:593)
[142248.500676] ____fput (/home/arter97/kernel/liquorix/fs/file_table.c:282)
[142248.500678] task_work_run (/home/arter97/kernel/liquorix/kernel/task_work.c:115 (discriminator 1))
[142248.500680] do_syscall_64 (/home/arter97/kernel/liquorix/include/linux/tracehook.h:188)
[142248.500682] entry_SYSCALL_64_after_hwframe (/home/arter97/kernel/liquorix/arch/x86/entry/entry_64.S:185)
[142248.500684] RIP: 0033:0x7fbf7363213b
[142248.500685] Code: 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 43 b9 f7 ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 91 b9 f7 ff 8b 44
All code
========
0: 03 00 add (%rax),%eax
2: 00 00 add %al,(%rax)
4: 0f 05 syscall
6: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
c: 77 41 ja 0x4f
e: c3 ret
f: 48 83 ec 18 sub $0x18,%rsp
13: 89 7c 24 0c mov %edi,0xc(%rsp)
17: e8 43 b9 f7 ff call 0xfffffffffff7b95f
1c: 8b 7c 24 0c mov 0xc(%rsp),%edi
20: 41 89 c0 mov %eax,%r8d
23: b8 03 00 00 00 mov $0x3,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 35 ja 0x67
32: 44 89 c7 mov %r8d,%edi
35: 89 44 24 0c mov %eax,0xc(%rsp)
39: e8 91 b9 f7 ff call 0xfffffffffff7b9cf
3e: 8b .byte 0x8b
3f: 44 rex.R
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 35 ja 0x3d
8: 44 89 c7 mov %r8d,%edi
b: 89 44 24 0c mov %eax,0xc(%rsp)
f: e8 91 b9 f7 ff call 0xfffffffffff7b9a5
14: 8b .byte 0x8b
15: 44 rex.R
[142248.500686] RSP: 002b:00007fbf72d0c390 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[142248.500687] RAX: 0000000000000000 RBX: 000055e4a1215750 RCX: 00007fbf7363213b
[142248.500688] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000000000009f
[142248.500689] RBP: 00007fbf72d0f770 R08: 0000000000000000 R09: 0000000000000000
[142248.500690] R10: 0000000000000000 R11: 0000000000000293 R12: 000055e4a1215c90
[142248.500691] R13: 0000000000000568 R14: 000055e4a113e980 R15: 000000000000009f
[142248.500692] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_set xt_geoip(O) xt_multiport ip_set_hash_ip ip_set nvme_fabrics xt_conntrack nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink zfs(PO) zunicode(PO) intel_rapl_msr zzstd(O) zlua(O) zavl(PO) intel_rapl_common icp(PO) btrfs x86_pkg_temp_thermal eeepc_wmi intel_powerclamp asus_wmi xor zcommon(PO) xfs raid6_pq coretemp znvpair(PO) sparse_keymap wmi_bmof intel_wmi_thunderbolt mxm_wmi spl(O) kvm_intel xhci_pci i40e mei_me i2c_i801 xhci_hcd ahci mei libahci wmi video acpi_pad mac_hid sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4
[142248.500711] CR2: 0000000000000078
[142248.500712] ---[ end trace 525f8b090c66add0 ]---
This is what I get. Unfortunately, by another dumb mistake, I had rm -rf uncommented on my build script so the obj files were missing. I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.
I'll build a new kernel with the latest trunk merged. Thanks.
I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.
It seems this is working fine :-)
Thank you for the output! It really looks like it is linked to e5755ac577dcc6685bfeb5e6d454db98bd7a7727. Do you mind if I close the ticket now and you re-open it if you still have the issue?
I had to build it again to get that info, but I'm not sure if rebuilding the same sources and same toolchain can change the output or not.
It seems this is working fine :-)
Thank you for the output! It really looks like it is linked to e5755ac. Do you mind if I close the ticket now and you re-open it if you still have the issue?
Not at all.
I'm quite surprised that this is triggered so late down the road. I've been using v5.4 with MPTCP for years at this point :)
Anyways, I'm using the latest kernel and everything seems fine now. I'll try to use it without reboots for weeks and report back if anything happens.
Thanks!
The bug happens in very specific conditions, that's maybe why :)
Uptime is 6 weeks now, no issues so far. Must be fixed :)
Not sure if this is an upstream breakage, but I've just noticed that my server had a NULL pointer dereference while running shadowsocks-rust with MPTCP.
As you can see in the timestamp, it occurred really late after the system's booted (MPTCP is being used right after the system's booted).
Unfortunately, I have frame pointers and unwinder disabled, but the guess unwinder seems to point at MPTCP.
Thanks.