search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
888 stars 335 forks source link

mptcp (with redundant schedular) crash #385

Closed gou4shi1 closed 2 years ago

gou4shi1 commented 4 years ago 
Apr  3 14:45:38 ecu kernel: [  253.406282] CPU: 1 PID: 16072 Comm: camera_stream_p Tainted: P           OE    4.15.18-weride-20200320 #4
Apr  3 14:45:38 ecu kernel: [  253.406286] Hardware name: Dell Inc. Precision 7920 Tower/060K5C, BIOS 2.1.4 07/29/2019
Apr  3 14:45:38 ecu kernel: [  253.406308] RIP: 0010:tcp_rbtree_insert+0x16/0x60
Apr  3 14:45:38 ecu kernel: [  253.406312] RSP: 0018:ffffb6f720577c30 EFLAGS: 00010282
Apr  3 14:45:38 ecu kernel: [  253.406318] RAX: ff98a49b2baf4040 RBX: ffff98a588390000 RCX: ffff98a474f1d200
Apr  3 14:45:38 ecu kernel: [  253.406323] RDX: ffff98a474f1d208 RSI: 000000000fceedc4 RDI: ffff989dd5410480
Apr  3 14:45:38 ecu kernel: [  253.406327] RBP: ffffb6f720577c30 R08: ffff98a588390150 R09: ffff989ddfc40000
Apr  3 14:45:38 ecu kernel: [  253.406331] R10: ffffb6f7205779e8 R11: 000000002b30059a R12: ffff989dd5410480
Apr  3 14:45:38 ecu kernel: [  253.406334] R13: 000000000000003a R14: ffff989d787a3180 R15: ffff989d76fe7380
Apr  3 14:45:38 ecu kernel: [  253.406340] FS:  00007fce217fa700(0000) GS:ffff989ddfc40000(0000) knlGS:0000000000000000
Apr  3 14:45:38 ecu kernel: [  253.406344] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr  3 14:45:38 ecu kernel: [  253.406348] CR2: 00007f858377a8e0 CR3: 00000007f5676001 CR4: 00000000007606e0
Apr  3 14:45:38 ecu kernel: [  253.406354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr  3 14:45:38 ecu kernel: [  253.406358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr  3 14:45:38 ecu kernel: [  253.406361] PKRU: 55555554
Apr  3 14:45:38 ecu kernel: [  253.406364] Call Trace:
Apr  3 14:45:38 ecu kernel: [  253.406378]  tcp_event_new_data_sent+0x54/0xa0
Apr  3 14:45:38 ecu kernel: [  253.406394]  mptcp_write_xmit+0x1fc/0x3e0
Apr  3 14:45:38 ecu kernel: [  253.406403]  tcp_push_one+0x4a/0x50
Apr  3 14:45:38 ecu kernel: [  253.406409]  tcp_sendmsg_locked+0x88d/0xff0
Apr  3 14:45:38 ecu kernel: [  253.406416]  tcp_sendmsg+0x2c/0x50
Apr  3 14:45:38 ecu kernel: [  253.406432]  inet_sendmsg+0x2e/0xb0
Apr  3 14:45:38 ecu kernel: [  253.406444]  sock_sendmsg+0x3e/0x50
Apr  3 14:45:38 ecu kernel: [  253.406449]  SYSC_sendto+0x13f/0x180
Apr  3 14:45:38 ecu kernel: [  253.406468]  ? __getnstimeofday64+0x43/0xd0
Apr  3 14:45:38 ecu kernel: [  253.406484]  ? _copy_to_user+0x26/0x30
Apr  3 14:45:38 ecu kernel: [  253.406496]  ? put_timespec64+0x3f/0x60
Apr  3 14:45:38 ecu kernel: [  253.406504]  SyS_sendto+0xe/0x10
Apr  3 14:45:38 ecu kernel: [  253.406516]  do_syscall_64+0x73/0x130
Apr  3 14:45:38 ecu kernel: [  253.406524]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Apr  3 14:45:38 ecu kernel: [  253.406529] RIP: 0033:0x7fcf38f779ff
Apr  3 14:45:38 ecu kernel: [  253.406531] RSP: 002b:00007fce217f9570 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Apr  3 14:45:38 ecu kernel: [  253.406535] RAX: ffffffffffffffda RBX: 000000000000001c RCX: 00007fcf38f779ff
Apr  3 14:45:38 ecu kernel: [  253.406538] RDX: 0000000000001eaa RSI: 00007fce08fed910 RDI: 000000000000001c
Apr  3 14:45:38 ecu kernel: [  253.406541] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Apr  3 14:45:38 ecu kernel: [  253.406544] R10: 0000000000004000 R11: 0000000000000246 R12: 00007fce08fed910
Apr  3 14:45:38 ecu kernel: [  253.406547] R13: 0000000000004000 R14: 0000000000001eaa R15: 0000000000000000
Apr  3 14:45:38 ecu kernel: [  253.406550] Code: 7b 18 00 74 92 0f 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 f8 31 c9 48 89 f7 4c 89 c2 48 89 e5 eb 15 <8b> 70 28 39 77 28 48 8d 48 10 48 8d 50 08 48 0f 48 d1 48 89 c1 
Apr  3 14:45:38 ecu kernel: [  253.406607] RIP: tcp_rbtree_insert+0x16/0x60 RSP: ffffb6f720577c30
Apr  3 14:45:38 ecu kernel: [  253.406611] ---[ end trace 97aeead56c70b4db ]---

I patched http://multipath-tcp.org/patches/mptcp-v4.15-69cdc26b88d2.patch and https://github.com/multipath-tcp/mptcp/commit/f6efc9d243997ea9c10430a56c3c0425417e53fa to https://launchpad.net/ubuntu/+source/linux/4.15.0-58.64

gou4shi1 commented 4 years ago

another crash:

Apr  3 15:24:42 ecu kernel: [  366.349901] CPU: 25 PID: 15650 Comm: camera_stream_p Tainted: P           OE    4.15.18-weride-20200320 #4
Apr  3 15:24:42 ecu kernel: [  366.349905] Hardware name: Dell Inc. Precision 7920 Tower/060K5C, BIOS 2.1.4 07/29/2019
Apr  3 15:24:42 ecu kernel: [  366.349914] RIP: 0010:tcp_fragment+0x285/0x3e0
Apr  3 15:24:42 ecu kernel: [  366.349918] RSP: 0018:ffffb8564f503bc8 EFLAGS: 00010246
Apr  3 15:24:42 ecu kernel: [  366.349923] RAX: 0000000000000001 RBX: ffff8fa87ba71f80 RCX: 0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.349928] RDX: 00000000ffffffff RSI: ffff8fa87ba71f80 RDI: ffff8fabd5d9c200
Apr  3 15:24:42 ecu kernel: [  366.349932] RBP: ffffb8564f503bf8 R08: 0000000000000007 R09: 0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.349937] R10: 0000000000000014 R11: ffff8fa41f806e80 R12: ffff8fabd5d9c200
Apr  3 15:24:42 ecu kernel: [  366.349941] R13: 0000000000000000 R14: 00000000ffffffff R15: ffff8fa3cccd86c0
Apr  3 15:24:42 ecu kernel: [  366.349947] FS:  00007f6dc8ff9700(0000) GS:ffff8fa41ff40000(0000) knlGS:0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.349951] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr  3 15:24:42 ecu kernel: [  366.349955] CR2: 0000000000000009 CR3: 0000001003d4c002 CR4: 00000000007606e0
Apr  3 15:24:42 ecu kernel: [  366.349960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.349964] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr  3 15:24:42 ecu kernel: [  366.349968] PKRU: 55555554
Apr  3 15:24:42 ecu kernel: [  366.349971] Call Trace:
Apr  3 15:24:42 ecu kernel: [  366.349985]  mptcp_fragment.constprop.19+0x53/0x120
Apr  3 15:24:42 ecu kernel: [  366.349991]  mptcp_write_xmit+0x178/0x3e0
Apr  3 15:24:42 ecu kernel: [  366.349999]  __tcp_push_pending_frames+0x40/0xe0
Apr  3 15:24:42 ecu kernel: [  366.350005]  tcp_push+0xdb/0x100
Apr  3 15:24:42 ecu kernel: [  366.350011]  tcp_sendmsg_locked+0x2f9/0xff0
Apr  3 15:24:42 ecu kernel: [  366.350018]  tcp_sendmsg+0x2c/0x50
Apr  3 15:24:42 ecu kernel: [  366.350026]  inet_sendmsg+0x2e/0xb0
Apr  3 15:24:42 ecu kernel: [  366.350035]  sock_sendmsg+0x3e/0x50
Apr  3 15:24:42 ecu kernel: [  366.350041]  SYSC_sendto+0x13f/0x180
Apr  3 15:24:42 ecu kernel: [  366.350053]  ? __getnstimeofday64+0x43/0xd0
Apr  3 15:24:42 ecu kernel: [  366.350063]  ? _copy_to_user+0x26/0x30
Apr  3 15:24:42 ecu kernel: [  366.350070]  ? put_timespec64+0x3f/0x60
Apr  3 15:24:42 ecu kernel: [  366.350078]  SyS_sendto+0xe/0x10
Apr  3 15:24:42 ecu kernel: [  366.350087]  do_syscall_64+0x73/0x130
Apr  3 15:24:42 ecu kernel: [  366.350096]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Apr  3 15:24:42 ecu kernel: [  366.350102] RIP: 0033:0x7f6edbec99ff
Apr  3 15:24:42 ecu kernel: [  366.350106] RSP: 002b:00007f6dc8ff8570 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Apr  3 15:24:42 ecu kernel: [  366.350112] RAX: ffffffffffffffda RBX: 000000000000001c RCX: 00007f6edbec99ff
Apr  3 15:24:42 ecu kernel: [  366.350116] RDX: 0000000000000a5e RSI: 00007f6da50e7fe0 RDI: 000000000000001c
Apr  3 15:24:42 ecu kernel: [  366.350120] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.350124] R10: 0000000000004000 R11: 0000000000000246 R12: 00007f6da50e7fe0
Apr  3 15:24:42 ecu kernel: [  366.350128] R13: 0000000000004000 R14: 0000000000000a5e R15: 0000000000000000
Apr  3 15:24:42 ecu kernel: [  366.350133] Code: 48 8d 4b 78 49 89 8f 80 00 00 00 48 89 53 78 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 03 49 89 5f 08 49 89 07 4c 89 3b <4c> 89 78 08 41 83 84 24 68 01 00 00 01 48 83 c4 08 5b 31 c0 41 
Apr  3 15:24:42 ecu kernel: [  366.350219] RIP: tcp_fragment+0x285/0x3e0 RSP: ffffb8564f503bc8
Apr  3 15:24:42 ecu kernel: [  366.350222] CR2: 0000000000000009
Apr  3 15:24:42 ecu kernel: [  366.350228] ---[ end trace c140d2ab3d5684f0 ]---
matttbe commented 4 years ago

Hi

We no longer support the kernel v4.15. The patch you mentioned is very old. There are maybe many missing fixes there.

Could it be possible for you to use the kernel v4.19 from our mptcp_v0.95 branch?

Maybe you will need additional patches for Ubuntu. In this page, you can find .deb files and .patch: https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.105/ I would suggest to first try the .deb you can find there (without MPTCP). If everything is OK on your side (hardware OK), then apply the patches you can find there in MPTCP repo. But it will be hard for us to help you with this v4.15 kernel.

gou4shi1 commented 4 years ago

@matttbe Hi I have switch to the mptcp_v0.95 branch (with some patches from https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.105) However, redundant scheduler still crash:

 Apr 15 15:39:49 ecu kernel: [ 1644.574252] perf: interrupt took too long (2524 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Apr 15 15:42:17 ecu kernel: [ 1792.948379] general protection fault: 0000 [#1] SMP PTI
Apr 15 15:42:17 ecu kernel: [ 1792.948387] CPU: 19 PID: 0 Comm: swapper/19 Tainted: P           OE     4.19.105-weride-20200413 #1
Apr 15 15:42:17 ecu kernel: [ 1792.948389] Hardware name: Dell Inc. Precision 7920 Tower/028PX1, BIOS 1.9.2 01/24/2019
Apr 15 15:42:17 ecu kernel: [ 1792.948398] RIP: 0010:rb_next+0x25/0x50
Apr 15 15:42:17 ecu kernel: [ 1792.948401] Code: 00 00 00 00 00 55 48 8b 0f 48 89 e5 48 39 cf 74 36 48 8b 47 08 48 85 c0 75 05 eb 1a 48 89 d0 48 8b 50 10 48 85 d2 75 f4 5d c3 <48> 3b 79 08 75 f8 48 8b 09 48 89 d7 48 83 e1 fc 48 85 c9 48 89 ca
Apr 15 15:42:17 ecu kernel: [ 1792.948403] RSP: 0018:ffff98af5f7c3a70 EFLAGS: 00010282
Apr 15 15:42:17 ecu kernel: [ 1792.948405] RAX: a5e469786f0a6f40 RBX: ffff98a6e5f6bd40 RCX: a5e469786f0a6f40
Apr 15 15:42:17 ecu kernel: [ 1792.948406] RDX: a5e469786f0a6f40 RSI: ffff98aa3c739f80 RDI: ffff98aa3c739f80
Apr 15 15:42:17 ecu kernel: [ 1792.948407] RBP: ffff98af5f7c3a70 R08: 0000000000000002 R09: ffff98a675e0bd80
Apr 15 15:42:17 ecu kernel: [ 1792.948409] R10: ffff98af5f7c3ad8 R11: 0000000000000001 R12: ffff98a6e5f6be98
Apr 15 15:42:17 ecu kernel: [ 1792.948410] R13: ffff98a671811fb8 R14: ffff98a675e0ad00 R15: ffff98a675e0ad00
Apr 15 15:42:17 ecu kernel: [ 1792.948412] FS:  0000000000000000(0000) GS:ffff98af5f7c0000(0000) knlGS:0000000000000000
Apr 15 15:42:17 ecu kernel: [ 1792.948413] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 15 15:42:17 ecu kernel: [ 1792.948415] CR2: 00007fb0a0cdc001 CR3: 00000004f860a004 CR4: 00000000007606e0
Apr 15 15:42:17 ecu kernel: [ 1792.948416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 15 15:42:17 ecu kernel: [ 1792.948417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 15 15:42:17 ecu kernel: [ 1792.948418] PKRU: 55555554
Apr 15 15:42:17 ecu kernel: [ 1792.948419] Call Trace:
Apr 15 15:42:17 ecu kernel: [ 1792.948421]  <IRQ>
Apr 15 15:42:17 ecu kernel: [ 1792.948429]  redsched_next_skb_from_queue+0x39/0x60
Apr 15 15:42:17 ecu kernel: [ 1792.948432]  mptcp_red_next_segment+0x130/0x300
Apr 15 15:42:17 ecu kernel: [ 1792.948436]  mptcp_write_xmit+0xc6/0x4b0
Apr 15 15:42:17 ecu kernel: [ 1792.948442]  __tcp_push_pending_frames+0x40/0xe0
Apr 15 15:42:17 ecu kernel: [ 1792.948445]  mptcp_write_space+0x43/0x50
Apr 15 15:42:17 ecu kernel: [ 1792.948448]  tcp_check_space+0x77/0xb0
Apr 15 15:42:17 ecu kernel: [ 1792.948450]  tcp_rcv_established+0x2b4/0x620

and the OS freeze immediately after this crash...

gou4shi1 commented 4 years ago

here is the kern.log before crash:

Apr 15 15:18:44 ecu kernel: [  380.252238] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 234
Apr 15 15:19:02 ecu kernel: [  397.884510] mptcp_alloc_mpcb: created mpcb with token 0x18cad5c1
Apr 15 15:19:02 ecu kernel: [  397.884515] mptcp_add_sock: token 0x18cad5c1 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:42578 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  397.884527] mptcp_alloc_mpcb: created mpcb with token 0x4126ec2d
Apr 15 15:19:02 ecu kernel: [  397.884529] mptcp_add_sock: token 0x4126ec2d pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:42578
Apr 15 15:19:02 ecu kernel: [  398.089848] mptcp_alloc_mpcb: created mpcb with token 0xdb33f4c8
Apr 15 15:19:02 ecu kernel: [  398.089855] mptcp_add_sock: token 0xdb33f4c8 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:42982 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.089871] mptcp_alloc_mpcb: created mpcb with token 0x8ce0ddb4
Apr 15 15:19:02 ecu kernel: [  398.089873] mptcp_add_sock: token 0x8ce0ddb4 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:42982
Apr 15 15:19:02 ecu kernel: [  398.118033] mptcp_alloc_mpcb: created mpcb with token 0x10218cdd
Apr 15 15:19:02 ecu kernel: [  398.118044] mptcp_add_sock: token 0x10218cdd pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43044 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.118065] mptcp_alloc_mpcb: created mpcb with token 0xee660a4a
Apr 15 15:19:02 ecu kernel: [  398.118067] mptcp_add_sock: token 0xee660a4a pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43044
Apr 15 15:19:02 ecu kernel: [  398.184879] mptcp_alloc_mpcb: created mpcb with token 0x967c2cdf
Apr 15 15:19:02 ecu kernel: [  398.184887] mptcp_add_sock: token 0x967c2cdf pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43182 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.184906] mptcp_alloc_mpcb: created mpcb with token 0xd41a32f1
Apr 15 15:19:02 ecu kernel: [  398.184909] mptcp_add_sock: token 0xd41a32f1 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43182
Apr 15 15:19:02 ecu kernel: [  398.221018] mptcp_alloc_mpcb: created mpcb with token 0xa162c761
Apr 15 15:19:02 ecu kernel: [  398.221025] mptcp_add_sock: token 0xa162c761 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43294 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.221041] mptcp_alloc_mpcb: created mpcb with token 0xcf94836d
Apr 15 15:19:02 ecu kernel: [  398.221043] mptcp_add_sock: token 0xcf94836d pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43294
Apr 15 15:19:02 ecu kernel: [  398.243590] mptcp_alloc_mpcb: created mpcb with token 0x3eb341f9
Apr 15 15:19:02 ecu kernel: [  398.243600] mptcp_add_sock: token 0x3eb341f9 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43344 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.243616] mptcp_alloc_mpcb: created mpcb with token 0x8b86c1f1
Apr 15 15:19:02 ecu kernel: [  398.243618] mptcp_add_sock: token 0x8b86c1f1 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43344
Apr 15 15:19:02 ecu kernel: [  398.274330] mptcp_alloc_mpcb: created mpcb with token 0x3ec2e45c
Apr 15 15:19:02 ecu kernel: [  398.274339] mptcp_add_sock: token 0x3ec2e45c pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43398 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.274358] mptcp_alloc_mpcb: created mpcb with token 0x77032541
Apr 15 15:19:02 ecu kernel: [  398.274361] mptcp_add_sock: token 0x77032541 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43398
Apr 15 15:19:02 ecu kernel: [  398.295094] mptcp_alloc_mpcb: created mpcb with token 0xdeb74fc8
Apr 15 15:19:02 ecu kernel: [  398.295103] mptcp_add_sock: token 0xdeb74fc8 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43446 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.295122] mptcp_alloc_mpcb: created mpcb with token 0x8b150316
Apr 15 15:19:02 ecu kernel: [  398.295125] mptcp_add_sock: token 0x8b150316 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43446
Apr 15 15:19:02 ecu kernel: [  398.299593] mptcp_alloc_mpcb: created mpcb with token 0xa10574c3
Apr 15 15:19:02 ecu kernel: [  398.299599] mptcp_add_sock: token 0xa10574c3 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43454 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:02 ecu kernel: [  398.299612] mptcp_alloc_mpcb: created mpcb with token 0xc4b575ba
Apr 15 15:19:02 ecu kernel: [  398.299614] mptcp_add_sock: token 0xc4b575ba pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43454
Apr 15 15:19:03 ecu kernel: [  398.383934] mptcp_alloc_mpcb: created mpcb with token 0x809618d2
Apr 15 15:19:03 ecu kernel: [  398.383939] mptcp_add_sock: token 0x809618d2 pi 1, src_addr:192.168.150.1:51308 dst_addr:218.107.32.5:5555
Apr 15 15:19:03 ecu kernel: [  398.395256] mptcp_alloc_mpcb: created mpcb with token 0xbf567afd
Apr 15 15:19:03 ecu kernel: [  398.395263] mptcp_add_sock: token 0xbf567afd pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43692 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.395279] mptcp_alloc_mpcb: created mpcb with token 0xb879d611
Apr 15 15:19:03 ecu kernel: [  398.395281] mptcp_add_sock: token 0xb879d611 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43692
Apr 15 15:19:03 ecu kernel: [  398.422110] mptcp_add_sock: token 0x809618d2 pi 2, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.422119] __mptcp_init4_subsockets: token 0x809618d2 pi 2 src_addr:192.168.4.127:0 dst_addr:218.107.32.5:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.422154] mptcp_add_sock: token 0x809618d2 pi 3, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.422158] __mptcp_init4_subsockets: token 0x809618d2 pi 3 src_addr:192.168.4.134:0 dst_addr:218.107.32.5:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.422184] mptcp_add_sock: token 0x809618d2 pi 4, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.422188] __mptcp_init4_subsockets: token 0x809618d2 pi 4 src_addr:192.168.4.127:0 dst_addr:172.17.0.1:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.422208] mptcp_add_sock: token 0x809618d2 pi 5, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.422212] __mptcp_init4_subsockets: token 0x809618d2 pi 5 src_addr:192.168.150.1:0 dst_addr:172.17.0.1:5555 ifidx: 7
Apr 15 15:19:03 ecu kernel: [  398.422235] mptcp_add_sock: token 0x809618d2 pi 6, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.422238] __mptcp_init4_subsockets: token 0x809618d2 pi 6 src_addr:192.168.4.134:0 dst_addr:172.17.0.1:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.483110] mptcp_alloc_mpcb: created mpcb with token 0x9999f2ea
Apr 15 15:19:03 ecu kernel: [  398.483117] mptcp_add_sock: token 0x9999f2ea pi 1, src_addr:192.168.150.1:51478 dst_addr:218.107.32.5:5555
Apr 15 15:19:03 ecu kernel: [  398.524011] mptcp_alloc_mpcb: created mpcb with token 0x42d68876
Apr 15 15:19:03 ecu kernel: [  398.524022] mptcp_add_sock: token 0x42d68876 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43964 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.524050] mptcp_alloc_mpcb: created mpcb with token 0x49c3117d
Apr 15 15:19:03 ecu kernel: [  398.524054] mptcp_add_sock: token 0x49c3117d pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:43964
Apr 15 15:19:03 ecu kernel: [  398.537916] mptcp_add_sock: token 0x9999f2ea pi 2, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.537925] __mptcp_init4_subsockets: token 0x9999f2ea pi 2 src_addr:192.168.4.127:0 dst_addr:218.107.32.5:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.537987] mptcp_add_sock: token 0x9999f2ea pi 3, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.537992] __mptcp_init4_subsockets: token 0x9999f2ea pi 3 src_addr:192.168.4.134:0 dst_addr:218.107.32.5:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.538019] mptcp_add_sock: token 0x9999f2ea pi 4, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.538022] __mptcp_init4_subsockets: token 0x9999f2ea pi 4 src_addr:192.168.4.127:0 dst_addr:172.17.0.1:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.538046] mptcp_add_sock: token 0x9999f2ea pi 5, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.538048] __mptcp_init4_subsockets: token 0x9999f2ea pi 5 src_addr:192.168.150.1:0 dst_addr:172.17.0.1:5555 ifidx: 7
Apr 15 15:19:03 ecu kernel: [  398.538074] mptcp_add_sock: token 0x9999f2ea pi 6, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.538077] __mptcp_init4_subsockets: token 0x9999f2ea pi 6 src_addr:192.168.4.134:0 dst_addr:172.17.0.1:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.547483] mptcp_alloc_mpcb: created mpcb with token 0x303e87ac
Apr 15 15:19:03 ecu kernel: [  398.547489] mptcp_add_sock: token 0x303e87ac pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44008 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.547505] mptcp_alloc_mpcb: created mpcb with token 0x31003773
Apr 15 15:19:03 ecu kernel: [  398.547507] mptcp_add_sock: token 0x31003773 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44008
Apr 15 15:19:03 ecu kernel: [  398.551995] mptcp_alloc_mpcb: created mpcb with token 0x2930e733
Apr 15 15:19:03 ecu kernel: [  398.552002] mptcp_add_sock: token 0x2930e733 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44018 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.552027] mptcp_alloc_mpcb: created mpcb with token 0x3f1f3e1b
Apr 15 15:19:03 ecu kernel: [  398.552030] mptcp_add_sock: token 0x3f1f3e1b pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44018
Apr 15 15:19:03 ecu kernel: [  398.598953] mptcp_alloc_mpcb: created mpcb with token 0x5129419
Apr 15 15:19:03 ecu kernel: [  398.598960] mptcp_add_sock: token 0x5129419 pi 1, src_addr:192.168.150.1:51760 dst_addr:218.107.32.5:5555
Apr 15 15:19:03 ecu kernel: [  398.648867] mptcp_add_sock: token 0x5129419 pi 2, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.648875] __mptcp_init4_subsockets: token 0x5129419 pi 2 src_addr:192.168.4.127:0 dst_addr:218.107.32.5:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.648908] mptcp_add_sock: token 0x5129419 pi 3, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.648912] __mptcp_init4_subsockets: token 0x5129419 pi 3 src_addr:192.168.4.134:0 dst_addr:218.107.32.5:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.648933] mptcp_add_sock: token 0x5129419 pi 4, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.648936] __mptcp_init4_subsockets: token 0x5129419 pi 4 src_addr:192.168.4.127:0 dst_addr:172.17.0.1:5555 ifidx: 6
Apr 15 15:19:03 ecu kernel: [  398.648951] mptcp_add_sock: token 0x5129419 pi 5, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.648954] __mptcp_init4_subsockets: token 0x5129419 pi 5 src_addr:192.168.150.1:0 dst_addr:172.17.0.1:5555 ifidx: 7
Apr 15 15:19:03 ecu kernel: [  398.648977] mptcp_add_sock: token 0x5129419 pi 6, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:03 ecu kernel: [  398.648979] __mptcp_init4_subsockets: token 0x5129419 pi 6 src_addr:192.168.4.134:0 dst_addr:172.17.0.1:5555 ifidx: 5
Apr 15 15:19:03 ecu kernel: [  398.727679] mptcp_alloc_mpcb: created mpcb with token 0xcfe3fa8d
Apr 15 15:19:03 ecu kernel: [  398.727691] mptcp_add_sock: token 0xcfe3fa8d pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44402 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.727709] mptcp_alloc_mpcb: created mpcb with token 0x91dbd2c1
Apr 15 15:19:03 ecu kernel: [  398.727711] mptcp_add_sock: token 0x91dbd2c1 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44402
Apr 15 15:19:03 ecu kernel: [  398.759864] mptcp_alloc_mpcb: created mpcb with token 0x1809c8cd
Apr 15 15:19:03 ecu kernel: [  398.759872] mptcp_add_sock: token 0x1809c8cd pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44460 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.759891] mptcp_alloc_mpcb: created mpcb with token 0xa5784d22
Apr 15 15:19:03 ecu kernel: [  398.759892] mptcp_add_sock: token 0xa5784d22 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44460
Apr 15 15:19:03 ecu kernel: [  398.773145] mptcp_alloc_mpcb: created mpcb with token 0x791ff9cc
Apr 15 15:19:03 ecu kernel: [  398.773153] mptcp_add_sock: token 0x791ff9cc pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44496 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.773179] mptcp_alloc_mpcb: created mpcb with token 0x49781221
Apr 15 15:19:03 ecu kernel: [  398.773182] mptcp_add_sock: token 0x49781221 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44496
Apr 15 15:19:03 ecu kernel: [  398.841719] mptcp_alloc_mpcb: created mpcb with token 0x71e79f59
Apr 15 15:19:03 ecu kernel: [  398.841728] mptcp_add_sock: token 0x71e79f59 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44640 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.841746] mptcp_alloc_mpcb: created mpcb with token 0x2df619f2
Apr 15 15:19:03 ecu kernel: [  398.841749] mptcp_add_sock: token 0x2df619f2 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44640
Apr 15 15:19:03 ecu kernel: [  398.851909] mptcp_alloc_mpcb: created mpcb with token 0xd02fafe8
Apr 15 15:19:03 ecu kernel: [  398.851916] mptcp_add_sock: token 0xd02fafe8 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44664 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  398.851931] mptcp_alloc_mpcb: created mpcb with token 0x62328c6c
Apr 15 15:19:03 ecu kernel: [  398.851934] mptcp_add_sock: token 0x62328c6c pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:44664
Apr 15 15:19:03 ecu kernel: [  399.141597] mptcp_alloc_mpcb: created mpcb with token 0x95a59e94
Apr 15 15:19:03 ecu kernel: [  399.141605] mptcp_add_sock: token 0x95a59e94 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:45368 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:03 ecu kernel: [  399.141623] mptcp_alloc_mpcb: created mpcb with token 0x72d51731
Apr 15 15:19:03 ecu kernel: [  399.141625] mptcp_add_sock: token 0x72d51731 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:45368
Apr 15 15:19:04 ecu kernel: [  399.389627] mptcp_alloc_mpcb: created mpcb with token 0x7e7e14c3
Apr 15 15:19:04 ecu kernel: [  399.389635] mptcp_add_sock: token 0x7e7e14c3 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:45982 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.389652] mptcp_alloc_mpcb: created mpcb with token 0xde56fafa
Apr 15 15:19:04 ecu kernel: [  399.389654] mptcp_add_sock: token 0xde56fafa pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:45982
Apr 15 15:19:04 ecu kernel: [  399.430758] mptcp_alloc_mpcb: created mpcb with token 0x83a9ba08
Apr 15 15:19:04 ecu kernel: [  399.430766] mptcp_add_sock: token 0x83a9ba08 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46026 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.430783] mptcp_alloc_mpcb: created mpcb with token 0xfc207457
Apr 15 15:19:04 ecu kernel: [  399.430785] mptcp_add_sock: token 0xfc207457 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46026
Apr 15 15:19:04 ecu kernel: [  399.504404] mptcp_alloc_mpcb: created mpcb with token 0x1f45d508
Apr 15 15:19:04 ecu kernel: [  399.504411] mptcp_add_sock: token 0x1f45d508 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46226 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.504427] mptcp_alloc_mpcb: created mpcb with token 0x1a7a260f
Apr 15 15:19:04 ecu kernel: [  399.504429] mptcp_add_sock: token 0x1a7a260f pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46226
Apr 15 15:19:04 ecu kernel: [  399.521668] mptcp_alloc_mpcb: created mpcb with token 0xbed3ebe9
Apr 15 15:19:04 ecu kernel: [  399.521675] mptcp_add_sock: token 0xbed3ebe9 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46274 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.521689] mptcp_alloc_mpcb: created mpcb with token 0xe99778e5
Apr 15 15:19:04 ecu kernel: [  399.521691] mptcp_add_sock: token 0xe99778e5 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46274
Apr 15 15:19:04 ecu kernel: [  399.544462] mptcp_alloc_mpcb: created mpcb with token 0x1df35cba
Apr 15 15:19:04 ecu kernel: [  399.544468] mptcp_add_sock: token 0x1df35cba pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46344 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.544482] mptcp_alloc_mpcb: created mpcb with token 0xc3b631f2
Apr 15 15:19:04 ecu kernel: [  399.544484] mptcp_add_sock: token 0xc3b631f2 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46344
Apr 15 15:19:04 ecu kernel: [  399.613635] mptcp_alloc_mpcb: created mpcb with token 0x97f55d42
Apr 15 15:19:04 ecu kernel: [  399.613647] mptcp_add_sock: token 0x97f55d42 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46564 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.613672] mptcp_alloc_mpcb: created mpcb with token 0xde0f70e4
Apr 15 15:19:04 ecu kernel: [  399.613676] mptcp_add_sock: token 0xde0f70e4 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46564
Apr 15 15:19:04 ecu kernel: [  399.651497] mptcp_alloc_mpcb: created mpcb with token 0xc42b6c74
Apr 15 15:19:04 ecu kernel: [  399.651504] mptcp_add_sock: token 0xc42b6c74 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46680 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:04 ecu kernel: [  399.651527] mptcp_alloc_mpcb: created mpcb with token 0x36b13c64
Apr 15 15:19:04 ecu kernel: [  399.651530] mptcp_add_sock: token 0x36b13c64 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:46680
Apr 15 15:19:05 ecu kernel: [  400.552197] mptcp_alloc_mpcb: created mpcb with token 0xd47ac00c
Apr 15 15:19:05 ecu kernel: [  400.552209] mptcp_add_sock: token 0xd47ac00c pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48538 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:05 ecu kernel: [  400.552238] mptcp_alloc_mpcb: created mpcb with token 0xbccfff93
Apr 15 15:19:05 ecu kernel: [  400.552242] mptcp_add_sock: token 0xbccfff93 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48538
Apr 15 15:19:05 ecu kernel: [  400.559624] mptcp_alloc_mpcb: created mpcb with token 0x9e2249ba
Apr 15 15:19:05 ecu kernel: [  400.559635] mptcp_add_sock: token 0x9e2249ba pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48550 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:05 ecu kernel: [  400.559660] mptcp_alloc_mpcb: created mpcb with token 0x549744e7
Apr 15 15:19:05 ecu kernel: [  400.559664] mptcp_add_sock: token 0x549744e7 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48550
Apr 15 15:19:05 ecu kernel: [  400.564071] mptcp_alloc_mpcb: created mpcb with token 0x63a3ba4a
Apr 15 15:19:05 ecu kernel: [  400.564081] mptcp_add_sock: token 0x63a3ba4a pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48558 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:05 ecu kernel: [  400.564104] mptcp_alloc_mpcb: created mpcb with token 0x5256d7a9
Apr 15 15:19:05 ecu kernel: [  400.564108] mptcp_add_sock: token 0x5256d7a9 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:48558
Apr 15 15:19:07 ecu kernel: [  402.667672] mptcp_alloc_mpcb: created mpcb with token 0xddd98fcf
Apr 15 15:19:07 ecu kernel: [  402.667736] mptcp_add_sock: token 0xddd98fcf pi 1, src_addr:192.168.150.1:40730 dst_addr:192.168.150.1:9003
Apr 15 15:19:07 ecu kernel: [  402.667887] mptcp_alloc_mpcb: created mpcb with token 0x23b5a180
Apr 15 15:19:07 ecu kernel: [  402.667925] mptcp_add_sock: token 0x23b5a180 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:c0a8:9601:9003 dst_addr:0000:0000:0000:0000:0000:ffff:c0a8:9601:40730
Apr 15 15:19:07 ecu kernel: [  402.668896] mptcp_add_sock: token 0xddd98fcf pi 2, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.668904] __mptcp_init4_subsockets: token 0xddd98fcf pi 2 src_addr:192.168.4.127:0 dst_addr:192.168.150.1:9003 ifidx: 6
Apr 15 15:19:07 ecu kernel: [  402.668944] mptcp_add_sock: token 0xddd98fcf pi 3, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.668948] __mptcp_init4_subsockets: token 0xddd98fcf pi 3 src_addr:192.168.4.134:0 dst_addr:192.168.150.1:9003 ifidx: 5
Apr 15 15:19:07 ecu kernel: [  402.668976] mptcp_add_sock: token 0xddd98fcf pi 4, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.668979] __mptcp_init4_subsockets: token 0xddd98fcf pi 4 src_addr:192.168.4.127:0 dst_addr:192.168.4.127:9003 ifidx: 6
Apr 15 15:19:07 ecu kernel: [  402.669038] mptcp_add_sock: token 0x23b5a180 pi 2, src_addr:0000:0000:0000:0000:0000:ffff:c0a8:047f:9003 dst_addr:0000:0000:0000:0000:0000:ffff:c0a8:047f:43177
Apr 15 15:19:07 ecu kernel: [  402.669059] mptcp_add_sock: token 0xddd98fcf pi 5, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.669062] __mptcp_init4_subsockets: token 0xddd98fcf pi 5 src_addr:192.168.150.1:0 dst_addr:192.168.4.127:9003 ifidx: 7
Apr 15 15:19:07 ecu kernel: [  402.669090] mptcp_add_sock: token 0xddd98fcf pi 6, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.669091] __mptcp_init4_subsockets: token 0xddd98fcf pi 6 src_addr:192.168.4.134:0 dst_addr:192.168.4.127:9003 ifidx: 5
Apr 15 15:19:07 ecu kernel: [  402.669124] mptcp_add_sock: token 0xddd98fcf pi 7, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.669126] __mptcp_init4_subsockets: token 0xddd98fcf pi 7 src_addr:192.168.4.127:0 dst_addr:192.168.4.134:9003 ifidx: 6
Apr 15 15:19:07 ecu kernel: [  402.669158] mptcp_add_sock: token 0xddd98fcf pi 8, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.669160] __mptcp_init4_subsockets: token 0xddd98fcf pi 8 src_addr:192.168.150.1:0 dst_addr:192.168.4.134:9003 ifidx: 7
Apr 15 15:19:07 ecu kernel: [  402.669185] mptcp_add_sock: token 0xddd98fcf pi 9, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:19:07 ecu kernel: [  402.669187] __mptcp_init4_subsockets: token 0xddd98fcf pi 9 src_addr:192.168.4.134:0 dst_addr:192.168.4.134:9003 ifidx: 5
Apr 15 15:19:07 ecu kernel: [  402.669222] mptcp_add_sock: token 0x23b5a180 pi 3, src_addr:0000:0000:0000:0000:0000:ffff:c0a8:0486:9003 dst_addr:0000:0000:0000:0000:0000:ffff:c0a8:0486:55575
Apr 15 15:19:10 ecu kernel: [  405.721362] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:7 state 7 is_meta? 0
Apr 15 15:19:10 ecu kernel: [  405.721401] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:6 state 7 is_meta? 0
Apr 15 15:19:14 ecu kernel: [  409.356592] mptcp_alloc_mpcb: created mpcb with token 0xa15009d
Apr 15 15:19:14 ecu kernel: [  409.356603] mptcp_add_sock: token 0xa15009d pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:56838 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:14 ecu kernel: [  409.356629] mptcp_alloc_mpcb: created mpcb with token 0xec8f1633
Apr 15 15:19:14 ecu kernel: [  409.356633] mptcp_add_sock: token 0xec8f1633 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:56838
Apr 15 15:19:28 ecu kernel: [  423.584590] pcan: PCAN-USB X6 CAN1: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:28 ecu kernel: [  423.987470] pcan: PCAN-USB X6 CAN2: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:28 ecu kernel: [  424.263491] pcan: PCAN-USB X6 CAN2: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:29 ecu kernel: [  424.699705] pcan: PCAN-USB X6 CAN2: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:29 ecu kernel: [  424.837927] pcan: PCAN-USB X6 CAN1: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:29 ecu kernel: [  424.851856] pcan: PCAN-USB X6 CAN2: msg type=1 id=701h l=8 lost: err -28 rxqsize=1000
Apr 15 15:19:32 ecu kernel: [  427.351331] mptcp_close: Close of meta_sk with tok 0x97f55d42
Apr 15 15:19:32 ecu kernel: [  427.351466] mptcp_del_sock: Removing subsock tok 0x97f55d42 pi:1 state 7 is_meta? 0
Apr 15 15:19:32 ecu kernel: [  427.351478] mptcp_del_sock: Removing subsock tok 0xde0f70e4 pi:1 state 7 is_meta? 0
Apr 15 15:19:32 ecu kernel: [  427.351479] mptcp_sock_destruct destroying meta-sk token 0x97f55d42
Apr 15 15:19:32 ecu kernel: [  427.351500] mptcp_close: Close of meta_sk with tok 0xde0f70e4
Apr 15 15:19:32 ecu kernel: [  427.351505] mptcp_sock_destruct destroying meta-sk token 0xde0f70e4
Apr 15 15:19:33 ecu kernel: [  429.139880] mptcp_alloc_mpcb: created mpcb with token 0x5b945dc
Apr 15 15:19:33 ecu kernel: [  429.139889] mptcp_add_sock: token 0x5b945dc pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:60748 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:19:33 ecu kernel: [  429.139910] mptcp_alloc_mpcb: created mpcb with token 0xeeb81526
Apr 15 15:19:33 ecu kernel: [  429.139913] mptcp_add_sock: token 0xeeb81526 pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:60748
Apr 15 15:19:34 ecu kernel: [  430.041061] mptcp_del_sock: Removing subsock tok 0x9999f2ea pi:6 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.041078] mptcp_del_sock: Removing subsock tok 0x9999f2ea pi:5 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.041083] mptcp_del_sock: Removing subsock tok 0x9999f2ea pi:4 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.041106] mptcp_del_sock: Removing subsock tok 0x5129419 pi:6 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.041127] mptcp_del_sock: Removing subsock tok 0x5129419 pi:5 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.041130] mptcp_del_sock: Removing subsock tok 0x5129419 pi:4 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.045475] mptcp_del_sock: Removing subsock tok 0x809618d2 pi:6 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.045484] mptcp_del_sock: Removing subsock tok 0x809618d2 pi:5 state 7 is_meta? 0
Apr 15 15:19:34 ecu kernel: [  430.045489] mptcp_del_sock: Removing subsock tok 0x809618d2 pi:4 state 7 is_meta? 0
Apr 15 15:19:38 ecu kernel: [  434.141029] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:8 state 7 is_meta? 0
Apr 15 15:19:38 ecu kernel: [  434.141065] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:5 state 7 is_meta? 0
Apr 15 15:19:38 ecu kernel: [  434.141077] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:2 state 7 is_meta? 0
Apr 15 15:19:38 ecu kernel: [  434.141086] mptcp_del_sock: Removing subsock tok 0xddd98fcf pi:3 state 7 is_meta? 0
Apr 15 15:22:05 ecu kernel: [  580.414475] mptcp_alloc_mpcb: created mpcb with token 0xdb9b83c1
Apr 15 15:22:05 ecu kernel: [  580.414486] mptcp_add_sock: token 0xdb9b83c1 pi 1, src_addr:192.168.150.1:46262 dst_addr:218.107.32.5:5555
Apr 15 15:22:05 ecu kernel: [  580.455780] mptcp_add_sock: token 0xdb9b83c1 pi 2, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:22:05 ecu kernel: [  580.455787] __mptcp_init4_subsockets: token 0xdb9b83c1 pi 2 src_addr:192.168.4.127:0 dst_addr:218.107.32.5:5555 ifidx: 6
Apr 15 15:22:05 ecu kernel: [  580.455824] mptcp_add_sock: token 0xdb9b83c1 pi 3, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:22:05 ecu kernel: [  580.455827] __mptcp_init4_subsockets: token 0xdb9b83c1 pi 3 src_addr:192.168.4.134:0 dst_addr:218.107.32.5:5555 ifidx: 5
Apr 15 15:22:05 ecu kernel: [  580.455844] mptcp_add_sock: token 0xdb9b83c1 pi 4, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:22:05 ecu kernel: [  580.455845] __mptcp_init4_subsockets: token 0xdb9b83c1 pi 4 src_addr:192.168.4.127:0 dst_addr:172.17.0.1:5555 ifidx: 6
Apr 15 15:22:05 ecu kernel: [  580.455865] mptcp_add_sock: token 0xdb9b83c1 pi 5, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:22:05 ecu kernel: [  580.455890] __mptcp_init4_subsockets: token 0xdb9b83c1 pi 5 src_addr:192.168.150.1:0 dst_addr:172.17.0.1:5555 ifidx: 7
Apr 15 15:22:05 ecu kernel: [  580.455911] mptcp_add_sock: token 0xdb9b83c1 pi 6, src_addr:0.0.0.0:0 dst_addr:0.0.0.0:0
Apr 15 15:22:05 ecu kernel: [  580.455914] __mptcp_init4_subsockets: token 0xdb9b83c1 pi 6 src_addr:192.168.4.134:0 dst_addr:172.17.0.1:5555 ifidx: 5
Apr 15 15:22:16 ecu kernel: [  591.675894] mptcp_alloc_mpcb: created mpcb with token 0xeb3a1ffa
Apr 15 15:22:16 ecu kernel: [  591.675902] mptcp_add_sock: token 0xeb3a1ffa pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:38958 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565
Apr 15 15:22:16 ecu kernel: [  591.675920] mptcp_alloc_mpcb: created mpcb with token 0x5f289c5f
Apr 15 15:22:16 ecu kernel: [  591.675923] mptcp_add_sock: token 0x5f289c5f pi 1, src_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:6565 dst_addr:0000:0000:0000:0000:0000:ffff:7f00:0001:38958
Apr 15 15:22:17 ecu kernel: [  592.951864] TCP: request_sock_TCP: Possible SYN flooding on port 35683. Sending cookies.  Check SNMP counters.
Apr 15 15:22:36 ecu kernel: [  612.055674] mptcp_del_sock: Removing subsock tok 0xdb9b83c1 pi:6 state 7 is_meta? 0
Apr 15 15:22:36 ecu kernel: [  612.055683] mptcp_del_sock: Removing subsock tok 0xdb9b83c1 pi:5 state 7 is_meta? 0
Apr 15 15:22:36 ecu kernel: [  612.055692] mptcp_del_sock: Removing subsock tok 0xdb9b83c1 pi:4 state 7 is_meta? 0

I have four 4G sim cards: 192.168.150.1, 192.168.4.127 and 192.168.4.134 218.107.32.5 is the address of server (4.19.55.mptcp installed from apt repo) 172.17.0.1 is the address of docker's NIC, our application run in docker

docker0: <NO-CARRIER,BROADCAST,MULTICAST,NOMULTIPATH,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b9:15:46:c1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

(I ran sudo ip link set dev docker0 multipath off)

I also have a VPN:

tun0: <POINTOPOINT,MULTICAST,NOMULTIPATH,NOARP,UP,LOWER_UP> mtu 1422 qdisc fq state UNKNOWN group default qlen 500
    link/none 
    inet 10.18.18.122/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::e1a2:b525:6537:5619/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

(I ran sudo ip link set dev tun0 multipath off)

$ ip rule show
0:      from all lookup local 
32762:  from 192.168.4.134 lookup enp213s0f2 
32763:  from 192.168.4.127 lookup enp213s0f3 
32764:  from 10.18.18.122 lookup tun0 
32765:  from 172.17.0.1 lookup docker0 
32766:  from all lookup main 
32767:  from all lookup default
gou4shi1 commented 4 years ago

Do I need to submit a crashdump file following https://multipath-tcp.org/pmwiki.php/Users/HandleCrashdumps?

gou4shi1 commented 4 years ago

here is the output of $ ./scripts/decode_stacktrace.sh vmlinux . /lib/modules/4.19.105-weride-20200415/ < crash.log

pr 15 15:42:17 ecu kernel: [ 1792.948379] general protection fault: 0000 [#1] SMP PTI
Apr 15 15:42:17 ecu kernel: [ 1792.948387] CPU: 19 PID: 0 Comm: swapper/19 Tainted: P           OE     4.19.105-weride-20200413 #1
Apr 15 15:42:17 ecu kernel: [ 1792.948389] Hardware name: Dell Inc. Precision 7920 Tower/028PX1, BIOS 1.9.2 01/24/2019
Apr 15 15:42:17 ecu kernel: [ 1792.948398] RIP: 0010:rb_next (??:?) 
Apr 15 15:42:17 ecu kernel: [ 1792.948401] Code: 00 00 00 00 00 55 48 8b 0f 48 89 e5 48 39 cf 74 36 48 8b 47 08 48 85 c0 75 05 eb 1a 48 89 d0 48 8b 50 10 48 85 d2 75 f4 5d c3 <48> 3b 79 08 75 f8 48 8b 09 48 89 d7 48 83 e1 fc 48 85 c9 48 89 ca
All code
========
   0:   00 00                   add    %al,(%rax)
   2:   00 00                   add    %al,(%rax)
   4:   00 55 48                add    %dl,0x48(%rbp)
   7:   8b 0f                   mov    (%rdi),%ecx
   9:   48 89 e5                mov    %rsp,%rbp
   c:   48 39 cf                cmp    %rcx,%rdi
   f:   74 36                   je     0x47
  11:   48 8b 47 08             mov    0x8(%rdi),%rax
  15:   48 85 c0                test   %rax,%rax
  18:   75 05                   jne    0x1f
  1a:   eb 1a                   jmp    0x36
  1c:   48 89 d0                mov    %rdx,%rax
  1f:   48 8b 50 10             mov    0x10(%rax),%rdx
  23:   48 85 d2                test   %rdx,%rdx
  26:   75 f4                   jne    0x1c
  28:   5d                      pop    %rbp
  29:   c3                      retq   
  2a:*  48 3b 79 08             cmp    0x8(%rcx),%rdi           <-- trapping instruction
  2e:   75 f8                   jne    0x28
  30:   48 8b 09                mov    (%rcx),%rcx
  33:   48 89 d7                mov    %rdx,%rdi
  36:   48 83 e1 fc             and    $0xfffffffffffffffc,%rcx
  3a:   48 85 c9                test   %rcx,%rcx
  3d:   48 89 ca                mov    %rcx,%rdx

Code starting with the faulting instruction
===========================================
   0:   48 3b 79 08             cmp    0x8(%rcx),%rdi
   4:   75 f8                   jne    0xfffffffffffffffe
   6:   48 8b 09                mov    (%rcx),%rcx
   9:   48 89 d7                mov    %rdx,%rdi
   c:   48 83 e1 fc             and    $0xfffffffffffffffc,%rcx
  10:   48 85 c9                test   %rcx,%rcx
  13:   48 89 ca                mov    %rcx,%rdx
Apr 15 15:42:17 ecu kernel: [ 1792.948403] RSP: 0018:ffff98af5f7c3a70 EFLAGS: 00010282
Apr 15 15:42:17 ecu kernel: [ 1792.948405] RAX: a5e469786f0a6f40 RBX: ffff98a6e5f6bd40 RCX: a5e469786f0a6f40
Apr 15 15:42:17 ecu kernel: [ 1792.948406] RDX: a5e469786f0a6f40 RSI: ffff98aa3c739f80 RDI: ffff98aa3c739f80
Apr 15 15:42:17 ecu kernel: [ 1792.948407] RBP: ffff98af5f7c3a70 R08: 0000000000000002 R09: ffff98a675e0bd80
Apr 15 15:42:17 ecu kernel: [ 1792.948409] R10: ffff98af5f7c3ad8 R11: 0000000000000001 R12: ffff98a6e5f6be98
Apr 15 15:42:17 ecu kernel: [ 1792.948410] R13: ffff98a671811fb8 R14: ffff98a675e0ad00 R15: ffff98a675e0ad00
Apr 15 15:42:17 ecu kernel: [ 1792.948412] FS:  0000000000000000(0000) GS:ffff98af5f7c0000(0000) knlGS:0000000000000000
Apr 15 15:42:17 ecu kernel: [ 1792.948413] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 15 15:42:17 ecu kernel: [ 1792.948415] CR2: 00007fb0a0cdc001 CR3: 00000004f860a004 CR4: 00000000007606e0
Apr 15 15:42:17 ecu kernel: [ 1792.948416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 15 15:42:17 ecu kernel: [ 1792.948417] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 15 15:42:17 ecu kernel: [ 1792.948418] PKRU: 55555554
Apr 15 15:42:17 ecu kernel: [ 1792.948419] Call Trace:
Apr 15 15:42:17 ecu kernel: [ 1792.948421]  <IRQ>
Apr 15 15:42:17 ecu kernel: [ 1792.948429] redsched_next_skb_from_queue (/home/dev/temp/mptcp/net/mptcp/mptcp_redundant.c:224) 
Apr 15 15:42:17 ecu kernel: [ 1792.948432] mptcp_red_next_segment (/home/dev/temp/mptcp/net/mptcp/mptcp_redundant.c:303) 
Apr 15 15:42:17 ecu kernel: [ 1792.948436] mptcp_write_xmit (/home/dev/temp/mptcp/net/mptcp/mptcp_output.c:766) 
Apr 15 15:42:17 ecu kernel: [ 1792.948442] __tcp_push_pending_frames (/home/dev/temp/mptcp/net/ipv4/tcp_output.c:2605) 
Apr 15 15:42:17 ecu kernel: [ 1792.948445] mptcp_write_space (/home/dev/temp/mptcp/net/mptcp/mptcp_output.c:905) 
Apr 15 15:42:17 ecu kernel: [ 1792.948448] tcp_check_space (/home/dev/temp/mptcp/net/ipv4/tcp_input.c:5225) 
Apr 15 15:42:17 ecu kernel: [ 1792.948450] tcp_rcv_established (/home/dev/temp/mptcp/net/ipv4/tcp_input.c:5303 /home/dev/temp/mptcp/net/ipv4/tcp_input.c:5742)

and the output of $ addr2line -e lib/rbtree.o rb_next+0x25 is lib/rbtree.c:428

gou4shi1 commented 4 years ago

Any one have suggestion on how to debug it?

gou4shi1 commented 4 years ago

I tried the 0.94 branch, crash at redundant_next_segment ……

cpaasch commented 4 years ago

I'm trying to repro the issue. No success yet.

Any hints as to how you can trigger it? E.g., are your interfaces flapping? Go up/down/... ? Do you need to send a lot of traffic? ...

cpaasch commented 4 years ago

Yes, I did a full run of my test-suite and am unable to repro and crashers. I think we need to narrow down the test-case here a bit :-)

cpaasch commented 4 years ago

Can you give the following patch a try:


commit a77e00990d7b88d62e0094eb04b947ee8d84137c
Author: Christoph Paasch <cpaasch@apple.com>
Date:   Tue Apr 21 16:09:32 2020 -0700

    mptcp: Make sure only valid skb's are pointed to by the redundant scheduler

    mptcp_restart_sending moves skbs from the rtx-queue to the send-queue.

    Ideally we would need to update the skb-pointer in the redundant
    scheduler. However, to be on the safe side, we can simply make sure that
    the skb is within snd_una -> snd_nxt.

    This needs to be backported to all the trees that have the rbnode usage
    in the retransmit-queue.

    Fixes: c61bc63e8f37 ("Merge tag 'v4.15-rc3' into mptcp_trunk")
    Signed-off-by: Christoph Paasch <cpaasch@apple.com>

diff --git a/net/mptcp/mptcp_redundant.c b/net/mptcp/mptcp_redundant.c
index 8226a1acf415..142c28e500b7 100644
--- a/net/mptcp/mptcp_redundant.c
+++ b/net/mptcp/mptcp_redundant.c
@@ -187,7 +187,9 @@ static void redsched_correct_skb_pointers(struct sock *meta_sk,
 {
    struct tcp_sock *meta_tp = tcp_sk(meta_sk);

-   if (red_p->skb && !after(red_p->skb_end_seq, meta_tp->snd_una))
+   if (red_p->skb &&
+       (!after(red_p->skb_end_seq, meta_tp->snd_una) ||
+        after(red_p->skb_end_seq, meta_tp->snd_nxt)))
        red_p->skb = NULL;
 }
gou4shi1 commented 4 years ago

Any hints as to how you can trigger it? E.g., are your interfaces flapping? Go up/down/... ?

Not flapping, after starting my application using mptcp, it will crash within 30min

Do you need to send a lot of traffic? ...

upload ~250KB/s

Can you give the following patch a try:

Sure, thx.

gou4shi1 commented 4 years ago

Can you give the following patch a try:

oops..it not works...

gou4shi1 commented 4 years ago

update: it works if I only enable 2 NICs, it crash if I enable 3 NICs

gou4shi1 commented 4 years ago

Yes, I did a full run of my test-suite and am unable to repro and crashers. I think we need to narrow down the test-case here a bit :-)

Does your test-suite contains the "redundant, 3 NICs" case? How can I run your test-suite?

cpaasch commented 4 years ago

update: it works if I only enable 2 NICs, it crash if I enable 3 NICs

Can you show the crashtrace? Thanks!

cpaasch commented 4 years ago

(wrong button 😇)

cpaasch commented 4 years ago

Yes, I did a full run of my test-suite and am unable to repro and crashers. I think we need to narrow down the test-case here a bit :-)

Does your test-suite contains the "redundant, 3 NICs" case? How can I run your test-suite?

It's this here: https://github.com/multipath-tcp/mptcp-scripts/tree/master/testing - has a mix of multi-interface tests,...

gou4shi1 commented 4 years ago 
[ 1076.523839] general protection fault: 0000 [#1] SMP PTI
[ 1076.523857] CPU: 7 PID: 21346 Comm: camera_pusher Kdump: loaded Tainted: P           OE     4.19.105-weride-20200423 #1
[ 1076.523860] Hardware name: Dell Inc. Precision 7920 Tower/028PX1, BIOS 1.9.2 01/24/2019
[ 1076.523876] RIP: 0010:rb_next+0x25/0x50
[ 1076.523880] Code: 00 00 00 00 00 55 48 8b 0f 48 89 e5 48 39 cf 74 36 48 8b 47 08 48 85 c0 75 05 eb 1a 48 89 d0 48 8b 50 10 48 85 d2 75 f4 5d c3 <48> 3b 79 08 75 f8 48 8b 09 48 89 d7 48 83 e1 fc 48 85 c9 48 89 ca
[ 1076.523883] RSP: 0000:ffff97835fbc3900 EFLAGS: 00010282
[ 1076.523886] RAX: a684c2b46b044c94 RBX: ffff978ad0b54ec0 RCX: a684c2b46b044c94
[ 1076.523888] RDX: a684c2b46b044c94 RSI: ffff978b57e64d80 RDI: ffff978b57e64d80
[ 1076.523889] RBP: ffff97835fbc3900 R08: 0000000000000002 R09: ffff9784dbbfdf80
[ 1076.523891] R10: ffff97835fbc3968 R11: 0000000000000001 R12: ffff978ad0b55018
[ 1076.523893] R13: ffff9783d98803b8 R14: ffff9784dbbff480 R15: ffff9784dbbff480
[ 1076.523895] FS:  00007f88abfff700(0000) GS:ffff97835fbc0000(0000) knlGS:0000000000000000
[ 1076.523897] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1076.523898] CR2: 000055f2db147000 CR3: 0000000fd081e004 CR4: 00000000007606e0
[ 1076.523900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1076.523901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1076.523902] PKRU: 55555554
[ 1076.523903] Call Trace:
[ 1076.523906]  <IRQ>
[ 1076.523916]  redsched_next_skb_from_queue+0x39/0x60
[ 1076.523920]  mptcp_red_next_segment+0x134/0x310
[ 1076.523923]  mptcp_write_xmit+0xc6/0x4b0
[ 1076.523930]  __tcp_push_pending_frames+0x40/0xe0
[ 1076.523933]  mptcp_data_ack+0x1ee/0xa50
[ 1076.523936]  mptcp_handle_options+0x12a/0x550
[ 1076.523939]  tcp_validate_incoming+0x379/0x460
[ 1076.523941]  tcp_rcv_established+0x1c4/0x620
[ 1076.523945]  tcp_v4_do_rcv+0xb8/0x240
[ 1076.523947]  tcp_v4_rcv+0xbb4/0xcb0
[ 1076.523953]  ip_local_deliver_finish+0x66/0x220
[ 1076.523955]  ip_local_deliver+0x7e/0xe0
[ 1076.523958]  ? ip_rcv_core.isra.19+0x2a0/0x2a0
[ 1076.523961]  ip_rcv_finish+0x84/0xa0
[ 1076.523963]  ip_rcv+0xbc/0xd0
[ 1076.523965]  ? ip_rcv_finish_core.isra.17+0x3a0/0x3a0
[ 1076.523977]  __netif_receive_skb_one_core+0x57/0x80
[ 1076.523979]  __netif_receive_skb+0x18/0x60
[ 1076.523982]  netif_receive_skb_internal+0x40/0xb0
[ 1076.523984]  napi_gro_receive+0xc5/0xf0
[ 1076.523996]  e1000_receive_skb+0x86/0xe0 [e1000e]
[ 1076.524002]  e1000_clean_rx_irq+0x1fe/0x3e0 [e1000e]
[ 1076.524008]  e1000e_poll+0x7e/0x2e0 [e1000e]
[ 1076.524013]  ? update_dl_rq_load_avg+0x10f/0x210
[ 1076.524015]  net_rx_action+0x140/0x3a0
[ 1076.524019]  __do_softirq+0xe4/0x2d4
[ 1076.524024]  irq_exit+0xc5/0xd0
[ 1076.524028]  do_IRQ+0x86/0xe0
[ 1076.524031]  common_interrupt+0xf/0xf
[ 1076.524032]  </IRQ>
[ 1076.524035] RIP: 0033:0x7f89fe353b70
[ 1076.524036] Code: e6 48 8b 95 a8 fd ff ff 49 f7 da eb 12 0f 1f 44 00 00 48 83 c0 04 49 39 c1 0f 86 cb f0 ff ff 0f b6 38 48 83 c6 02 41 83 c0 01 <40> d0 ef 40 00 3a 48 83 c2 01 0f b6 78 01 40 88 7e fe 0f b6 78 02
[ 1076.524038] RSP: 002b:00007f88abffe4c0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[ 1076.524039] RAX: 000055f52fe0bdbc RBX: 0000000000000640 RCX: 00007f88890d69ef
[ 1076.524041] RDX: 00007f88890616ef RSI: 00007f8888fa1d50 RDI: 0000000000000077
[ 1076.524042] RBP: 00007f88abffe740 R08: 0000000000000320 R09: 000055f52fed24c0
[ 1076.524043] R10: fffffffffffffce0 R11: 00007f89c64c4f90 R12: 00007f8888e304d0
[ 1076.524044] R13: 000055f52a359740 R14: 0000000000000011 R15: 0000000000000000

@cpaasch seems same before your patch for the full log, refer to https://gist.github.com/gou4shi1/1824a8d5f12e7f468a05a82668b6cb2c

gou4shi1 commented 4 years ago

I have the corresponding corefile, how can I send it to you?

cpaasch commented 4 years ago

Can you upload it to dropbox or any other file-sharing?

gou4shi1 commented 4 years ago

@cpaasch sorry for the late reply https://send.firefox.com/download/3b9ad8a1536ed241/#1Pv5FMU2zAXwmDCRhwB4Pg

matttbe commented 4 years ago

The last call trace looks linked to #392

@gou4shi1 may you try to reproduce the issue after having applied the patch available in #391 ?

gou4shi1 commented 4 years ago

@matttbe I tried (origin/mptcp_v0.95, 1afee53), it still crash immediately if I enable 3 NIC.

[ 2954.784578] general protection fault: 0000 [#1] SMP PTI
[ 2954.784586] CPU: 41 PID: 0 Comm: swapper/41 Kdump: loaded Tainted: P           OE     4.19.117-weride-200516 #1
[ 2954.784588] Hardware name: Dell Inc. Precision 7920 Tower/028PX1, BIOS 1.9.2 01/24/2019
[ 2954.784598] RIP: 0010:rb_next+0x25/0x50
[ 2954.784600] Code: 00 00 00 00 00 55 48 8b 0f 48 89 e5 48 39 cf 74 36 48 8b 47 08 48 85 c0 75 05 eb 1a 48 89 d0 48 8b 50 10 48 85 d2 75 f4 5d c3 <48> 3b 79 08 75 f8 48 8b 09 48 89 d7 48 83 e1 fc 48 85 c9 48 89 ca
[ 2954.784613] RSP: 0018:ffff977a9fa439a8 EFLAGS: 00010202
[ 2954.784615] RAX: 5bebd70017bf61b0 RBX: ffff976d106da300 RCX: 5bebd70017bf61b0
[ 2954.784617] RDX: 5bebd70017bf61b0 RSI: ffff977254107600 RDI: ffff977254107600
[ 2954.784618] RBP: ffff977a9fa439a8 R08: 000000000000002c R09: ffff9778bec87000
[ 2954.784619] R10: ffff977a9fa43a10 R11: 0000000000000001 R12: ffff976d106da458
[ 2954.784621] R13: ffff977a9a3006c0 R14: ffff976ebdf7b000 R15: ffff976ebdf7b000
[ 2954.784623] FS:  0000000000000000(0000) GS:ffff977a9fa40000(0000) knlGS:0000000000000000
[ 2954.784624] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2954.784626] CR2: 00007fc49c004768 CR3: 000000066c20a002 CR4: 00000000007606e0
[ 2954.784628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2954.784629] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2954.784630] PKRU: 55555554
[ 2954.784631] Call Trace:
[ 2954.784634]  <IRQ>
[ 2954.784643]  redsched_next_skb_from_queue+0x39/0x70
[ 2954.784646]  mptcp_red_next_segment+0x140/0x320
[ 2954.784649]  mptcp_write_xmit+0xc0/0x480
[ 2954.784655]  __tcp_push_pending_frames+0x40/0xe0
[ 2954.784658]  mptcp_process_data_ack+0x1f0/0xa50
[ 2954.784660]  mptcp_handle_options+0x12d/0x540
[ 2954.784663]  tcp_validate_incoming+0x379/0x460
[ 2954.784665]  tcp_rcv_established+0x1c4/0x620
[ 2954.784667]  tcp_v4_do_rcv+0xb8/0x240
[ 2954.784669]  tcp_v4_rcv+0xbb4/0xcb0
[ 2954.784673]  ip_local_deliver_finish+0x66/0x220
[ 2954.784675]  ip_local_deliver+0x7e/0xe0
[ 2954.784678]  ? ip_rcv_core.isra.19+0x2a0/0x2a0
[ 2954.784679]  ip_rcv_finish+0x84/0xa0
[ 2954.784681]  ip_rcv+0xbc/0xd0
[ 2954.784683]  ? ip_rcv_finish_core.isra.17+0x3a0/0x3a0
[ 2954.784687]  __netif_receive_skb_one_core+0x57/0x80
[ 2954.784689]  __netif_receive_skb+0x18/0x60
[ 2954.784691]  netif_receive_skb_internal+0x40/0xb0
[ 2954.784693]  napi_gro_receive+0xc5/0xf0
[ 2954.784702]  igb_poll+0x477/0xea0 [igb]
[ 2954.784705]  ? ttwu_do_activate+0x77/0x80
[ 2954.784707]  net_rx_action+0x140/0x3a0
[ 2954.784709]  __do_softirq+0xe4/0x2d4
[ 2954.784712]  irq_exit+0xc5/0xd0
[ 2954.784713]  do_IRQ+0x86/0xe0
[ 2954.784716]  common_interrupt+0xf/0xf
[ 2954.784717]  </IRQ>
[ 2954.784722] RIP: 0010:cpuidle_enter_state+0xa8/0x320
[ 2954.784723] Code: 3d 0d d6 7a 6a e8 28 98 86 ff 49 89 c4 0f 1f 44 00 00 31 ff e8 59 a3 86 ff 80 7d d3 00 0f 85 21 02 00 00 fb 66 0f 1f 44 00 00 <4d> 29 fc 48 ba cf f7 53 e3 a5 9b c4 20 4c 89 e0 49 c1 fc 3f 48 f7
[ 2954.784724] RSP: 0018:ffffb9c58659fe58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb
[ 2954.784725] RAX: ffff977a9fa62180 RBX: ffffd9c57fc44118 RCX: 000000000000001f
[ 2954.784726] RDX: 000002aff6e22e2e RSI: 000000002ac4545d RDI: 0000000000000000
[ 2954.784727] RBP: ffffb9c58659fe90 R08: 0000000000000002 R09: 0000000000021a40
[ 2954.784728] R10: ffffb9c58659fe28 R11: 0000000000000071 R12: 000002aff6e22e2e
[ 2954.784729] R13: 0000000000000002 R14: ffffffff96790eb8 R15: 000002aff6e1bf9e
[ 2954.784731]  ? cpuidle_enter_state+0x97/0x320
[ 2954.784733]  cpuidle_enter+0x17/0x20
[ 2954.784735]  call_cpuidle+0x23/0x40
[ 2954.784736]  do_idle+0x1f6/0x270
[ 2954.784738]  cpu_startup_entry+0x73/0x80
[ 2954.784741]  start_secondary+0x1ab/0x200
[ 2954.784744]  secondary_startup_64+0xa4/0xb0
[ 2954.784746] Modules linked in: nvidia_uvm(OE) pps_ldisc nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter bridge stp llc nls_utf8 cifs ccm fscache overlay intel_rapl skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel usb_serial_simple usbserial kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper snd_hda_codec_hdmi intel_cstate snd_hda_codec_realtek snd_hda_codec_generic nvidia_drm(POE) nvidia_modeset(POE) ipt_MASQUERADE xt_addrtype iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter ip_tables x_tables dell_smm_hwmon intel_rapl_perf dell_wmi snd_usb_audio snd_hda_intel dell_smbios snd_usbmidi_lib snd_hda_codec dcdbas snd_seq_midi sparse_keymap
[ 2954.784775]  snd_seq_midi_event snd_hda_core intel_wmi_thunderbolt nvidia(POE) pcan(OE) video wmi_bmof dell_wmi_descriptor snd_hwdep snd_rawmidi pcmcia snd_pcm pcmcia_core drm_kms_helper snd_seq drm ipmi_devintf snd_seq_device ipmi_msghandler snd_timer fb_sys_fops syscopyarea snd sysfillrect mei_me sysimgblt soundcore lpc_ich mei ioatdma mac_hid binfmt_misc tcp_bbr sch_fq mptcp_wvegas parport_pc ppdev lp parport autofs4 hid_led hid_generic usbhid hid uas usb_storage ixgbe igb ahci dca mdio e1000e i2c_algo_bit pata_acpi libahci wmi
cpaasch commented 4 years ago

@cpaasch sorry for the late reply https://send.firefox.com/download/3b9ad8a1536ed241/#1Pv5FMU2zAXwmDCRhwB4Pg

Sorry, I'm not able to download this file. I tried Safari & Chrome. None worked. Any way you can share it in another way? :)

cpaasch commented 4 years ago

Also, I would need the vmlinux file as well. Thanks :)

matttbe commented 2 years ago

A new fix has been applied. Does it fix this issue?

See commit: d1c53d9c42fc76199ce1e4a4b52428ae9ee6b87a (mptcp_v0.95 branch)

Please re-open this ticket if not.