syzkaller: WARNING in `__mptcp_move_skbs_from_subflow`

[cpaasch]

syzkaller-ID: 7bc336bf049b8e3d7efa860b71901d5e094b33ac

HEAD: de796451201b (07/11)

Trace:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 30694 at net/mptcp/protocol.c:703 __mptcp_move_skbs_from_subflow+0xc91/0xcd0
Modules linked in:
CPU: 0 PID: 30694 Comm: syz-executor.4 Not tainted 6.4.0-gde796451201b #28
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
RIP: 0010:__mptcp_move_skbs_from_subflow+0xc91/0xcd0 net/mptcp/protocol.c:703
Code: 04 25 28 00 00 00 48 3b 44 24 78 75 4a 89 e8 48 81 c4 80 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc e8 af cf 98 fe <0f> 0b eb b0 e8 a6 cf 98 fe 41 89 9e 18 01 00 00 41 89 df e9 da f3
RSP: 0018:ffffc90000003a48 EFLAGS: 00010246
RAX: ffffffff8285cf51 RBX: 00000000000003c4 RCX: ffff888138e40000
RDX: 0000000080000303 RSI: 000000000000ffa0 RDI: 00000000000003c4
RBP: ffff888131937af0 R08: ffffffff8285c903 R09: ffffffff82864143
R10: 0000000000000004 R11: ffffffff81d308f0 R12: 000000000000ffa0
R13: 0000000000068000 R14: ffff888131937af0 R15: ffff888131937af0
FS:  00007f6690962640(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020008000 CR3: 0000000133175001 CR4: 0000000000170ef0
Call Trace:
 <IRQ>
 mptcp_data_ready+0xdc/0x2d0 net/mptcp/protocol.c:777
 tcp_data_queue+0xc66/0x1e50 net/ipv4/tcp_input.c:5086
 tcp_rcv_established+0x638/0xae0 net/ipv4/tcp_input.c:6025
 tcp_v6_do_rcv+0x38c/0x870 net/ipv6/tcp_ipv6.c:1484
 tcp_v6_rcv+0x10c2/0x12e0 net/ipv6/tcp_ipv6.c:1745
 ip6_protocol_deliver_rcu+0x637/0xba0 net/ipv6/ip6_input.c:437
 ip6_input+0x82/0x150 net/ipv6/ip6_input.c:482
 ipv6_rcv+0x5d/0x120 include/linux/netfilter.h:303
 __netif_receive_skb+0x84/0x210 net/core/dev.c:5452
 process_backlog+0x12e/0x200 net/core/dev.c:5894
 __napi_poll+0x46/0x2e0 net/core/dev.c:6460
 napi_poll net/core/dev.c:6527 [inline]
 net_rx_action+0x1dd/0x460 net/core/dev.c:6660
 __do_softirq+0xed/0x2bd kernel/softirq.c:553
 do_softirq+0x7a/0xc0 kernel/softirq.c:454
 </IRQ>
 <TASK>
 __local_bh_enable_ip+0x7b/0x80 kernel/softirq.c:381
 rcu_read_unlock_bh include/linux/rcupdate.h:819 [inline]
 __dev_queue_xmit+0x821/0x17b0 net/core/dev.c:4230
 dev_queue_xmit include/linux/netdevice.h:3088 [inline]
 neigh_hh_output include/net/neighbour.h:528 [inline]
 neigh_output include/net/neighbour.h:542 [inline]
 ip6_finish_output2+0x6b0/0x8e0 net/ipv6/ip6_output.c:135
 ip6_output+0x9b/0x190 include/linux/netfilter.h:292
 ip6_xmit+0x608/0x8d0 include/net/dst.h:458
 inet6_csk_xmit+0x136/0x170 net/ipv6/inet6_connection_sock.c:135
 __tcp_transmit_skb+0xb69/0xf80 net/ipv4/tcp_output.c:1401
 tcp_transmit_skb net/ipv4/tcp_output.c:1419 [inline]
 tcp_write_xmit+0xb59/0x1d90 net/ipv4/tcp_output.c:2735
 __tcp_push_pending_frames+0x54/0x130 net/ipv4/tcp_output.c:2919
 mptcp_push_release net/mptcp/protocol.c:1447 [inline]
 __mptcp_push_pending+0x27f/0x3b0 net/mptcp/protocol.c:1582
 mptcp_sendmsg+0x6b0/0x830 net/mptcp/protocol.c:1837
 __sys_sendto+0x22f/0x360 net/socket.c:725
 __do_sys_sendto net/socket.c:2146 [inline]
 __se_sys_sendto net/socket.c:2142 [inline]
 __x64_sys_sendto+0x28/0x30 net/socket.c:2142
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x47/0xa0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x6e/0xd8
RIP: 0033:0x7f66916556a9
Code: 5c c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4f 37 0d 00 f7 d8 64 89 01 48
RSP: 002b:00007f6690961cd8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00000000006bc050 RCX: 00007f66916556a9
RDX: 0000000000007fdc RSI: 0000000020000040 RDI: 0000000000000004
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006bc05c
R13: fffffffffffffea8 R14: 00000000006bc050 R15: 000000000001fe40
 </TASK>
---[ end trace 0000000000000000 ]---
------------[ cut here ]------------

No reproducer.

Kconfig: Kconfig_k7_clean.txt

[pabeni]

Have you observed this one multiple times? if so perhaps we could add some debug patch in the export branch to dump possibly relevant additional info. Otherwise I have no idea :/

[cpaasch]

Haven't seen this one in a long while. We can close it!

[pabeni]

While investigating #437, I noticed this:

https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_ipv4.c#L1861

that is, packets enqueued to subflows backlog did not check for mptcp-constraints before coalescing. The subflows socket lock is almost never acquired by the user-space, so the race is very hard to reproduce, but in theory multiple packets carrying separate DSS could land into the subflow backlog and being coalesced ignoring the DSS info - basically dropping the newer's packet DSS - and thus triggering the splat reported above.

We could address the issue introducing an additional mptcp_skb_can_collapse(skb, tail) check in tcp_add_backlog(), but first I think it would be better to try to validate the above with a pktdrill test - we may need an new debug sockopt to keep the given subflow socket lock acquire for a specified time.