search

multipath-tcp / mptcp_net-next

Development version of the Upstream MultiPath TCP Linux kernel 🐧
https://mptcp.dev
Other
285 stars 41 forks source link

MPTCP connections are hanging when all subflows have been closed by timeout #430

Closed matttbe closed 1 year ago

matttbe commented 1 year ago

A server running rsyncd processes was accidentally rebooted while multiple "client" hosts had active rsync processes to it. On nearly all those hosts, rsync was in the state described in ticket #429 - the tcp connections were gone but the mptcp connection was hanging around and holding the rsync command open.

I also had one client rsync from exactly the same time stuck in the initial connect and MPTCP-SYN was the only entry in the output of "ss".

I'm going to try run some more tests killing the server's rsyncd processes and/or rebooting servers to see if it is reproducible.

Again, this is not normal operation for us and may be a different issue to the one described in #429 (but with the same result)?

_Originally posted by @daire-byrne in https://github.com/multipath-tcp/mptcp_net-next/issues/429#issuecomment-1691398398_

daire-byrne commented 1 year ago

Yea, so I think I have an idea what is happening here....

I tested lots of variations of reboots and killing the server process in mid transfer but nothing could reproduce it. Then I started playing around with the systemd scripts that setup all the routes and mptcp endpoints on boot.

While doing this, I believe I was able to reproduce the hang - I think maybe we have a race condition in the startup of the network, rsyncd and the script we use to setup the mptcp endpoints and routing.

Long story short, it looks like downing an interface (which is fine) and bringing it back up (without the routes) is possibly what caused the hangs in this case? I need to do some more tests and get the order of things right but this may not be a mptcp issue at all (unless it's still supposed to recover or fail gracefully).

I'll report back tomorrow.

daire-byrne commented 1 year ago

Okay, I'm not 100% sure this is the exact way to reproduce the problem we saw when a server was rebooted, but it's a replication of an rsync hang nonetheless.

Our servers use a systemd service script to setup the mptcp endpoints and routing on boot. This was missing a related dependency on the rsyncd systemd service so the ordering of start/stop is not guaranteed. Both might even start at the same time?

So let's setup the mptcp networking on each server as we do in production with the systemd script:

# serverA
ip mptcp endpoint flush
ip mptcp endpoint add 10.13.20.251 id 1 backup dev ens192
ip mptcp endpoint add 10.13.21.251 id 2 signal dev ens224
ip mptcp endpoint add 10.13.22.251 id 3 signal dev ens256
ip mptcp limits set subflow 4 add_addr_accepted 4

ip route add 10.17.21.251 via 10.13.21.1 dev ens224
ip route add 10.17.22.251 via 10.13.22.1 dev ens256

And then the same but reverse routes on serverB:

ip mptcp endpoint flush
ip mptcp endpoint add 10.17.20.251 id 1 backup dev ens192
ip mptcp endpoint add 10.17.21.251 id 2 signal dev ens224
ip mptcp endpoint add 10.17.22.251 id 3 signal dev ens256
ip mptcp limits set subflow 4 add_addr_accepted 4

ip route add 10.13.21.251 via 10.17.21.1 dev ens224
ip route add 10.13.22.251 via 10.17.22.1 dev ens256

And now lets do an rsync using serverA as the client to rsyncd running on serverB:

serverA # mptcpize run rsync -av 10.17.20.251::root/usr /tmp/testing

Everything works as expected and we see data being transferred between ens224 & ens256 interface pairs only.

Now down ens256 on serverB:

serverB # ifconfig ens256 down

The rsync continues uninterrupted as expected across the ens224 pair only and the mptcp endpoints are still defined for all interfaces. Now bring ens256 back up, but note that the route between the ens256 interfaces is obviously gone from serverB:

serverB # ifconfig ens256 up
serverB # ip route                 
default via 10.17.20.1 dev ens192 proto dhcp src 10.17.20.251 metric 100 
default via 10.17.21.1 dev ens224 proto static metric 101 
default via 10.17.22.1 dev ens256 proto static metric 102 
10.13.21.251 via 10.17.21.1 dev ens224 
{similar route for ens256 missing from here!}
10.17.20.0/24 dev ens192 proto kernel scope link src 10.17.20.251 metric 100 
10.17.21.0/24 dev ens224 proto kernel scope link src 10.17.21.251 metric 101 
10.17.22.0/24 dev ens256 proto kernel scope link src 10.17.22.251 metric 102

At this point the rsync client process on serverA is hung and the rsyncd process on rsyncd is also hung. The tcp connections are no longer reported in the output of "ss" but we see the "mptcp ESTAB" line:

serverB:
mptcp ESTAB 0      0       10.17.20.251:873   10.13.20.251:38872 users:(("rsync",pid=5280,fd=6)) ino:76504 sk:3005 cgroup:unreachable:1 <->
     skmem:(r0,rb6409000,t0,tb12833280,f3769,w8184135,o0,bl0,d0) add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:90f55862 write_seq:b5d9ad874a5e61ba snd_una:b5d9ad8749e1a939 rcv_nxt:ed721a5a706415

Now I can kill either the rsync client on serverA or the rsyncd process on serverB but the other end's process will never die until manually killed too.

I can add the correct route back into serverB for ens256, but the hung rsync processes never recover. All new rsync processes will work fine as expected.

So, I think that something like this is happening where the bringup or tear down of interfaces and routes during reboot, hung lots of client rsyncs that were running on different remote hosts.

Now I don't know if this is really an mptcp issue per say, but I'll have to think about how to improve the ordering of the network services and bringup and tear down steps to avoid such hangs.

pabeni commented 1 year ago

I could reproduce some of what you report above using a simplified, netns-based test-case. 

#!/bin/sh

DEVS="veth0 veth1 veth2"

for n in cl router srv; do
        ip netns del $n
        ip netns add $n
        ip -n $n link set dev lo up
        ip netns exec $n sysctl -w net.ipv4.conf.all.rp_filter=0
        ip netns exec $n sysctl -w net.ipv4.conf.default.rp_filter=0
        ip netns exec $n sysctl -w net.ipv4.conf.lo.rp_filter=0
        ip -n $n mptcp limits set add_addr_accepted 3 subflow 3
done

NET=1
for dev in $DEVS; do
        HOST=1
        for n in cl srv; do
                ip link add netns $n name $dev type veth  peer name "$dev"_"$n" netns router
                ip -n $n link set dev $dev up
                ip -n router link set dev "$dev"_"$n" up
                ip netns exec $n tc qdisc replace dev $dev root netem rate 10mbit
                ip -n $n addr add dev $dev 10.$NET.$HOST.1/24
                ip -n router addr add dev "$dev"_"$n" 10.$NET.$HOST.2/24

                if [ $dev = veth0 ]; then
                        ip -n $n mptcp endpoint add dev $dev 10.$NET.$HOST.1 backup
                else
                        ip -n $n mptcp endpoint add dev $dev 10.$NET.$HOST.1 signal
                        ip -n $n route add dev $dev 10.$NET.$((3 - $HOST)).1 via 10.$NET.$HOST.2
                fi
                ip -n $n route add default via 10.$NET.$HOST.2 dev $dev metric $((100 + $NET ))
                HOST=$((HOST+1))
        done
        NET=$((NET+1))
done

ip netns exec router sysctl -w net.ipv4.ip_forward=1

ip netns exec srv mptcpize run iperf3 -s -i 10 &
sleep 0.2
ip netns exec cl mptcpize run iperf3 -c 10.2.2.1 -i i -t 1000 &
sleep 20
ip -n cl link set dev veth2 down
sleep 20
ip -n cl link set dev veth2 up

but with the above the tcp-level subflows are still open and the transfer is not interrupted.

I instead reproduced the mptcp-level hangup with this packetdrill:

// This test case covers: https://github.com/multipath-tcp/mptcp_net-next/issues/430
// ensure that a tcp-level timeout is catched
--tolerance_usecs=100000
`../common/defaults.sh
 ../common/set_sysctls.py                       \
        /proc/sys/net/ipv4/tcp_syn_retries=1    \
        /proc/sys/net/ipv4/tcp_syn_linear_timeouts=0
`

0.0     socket(..., SOCK_STREAM, IPPROTO_MPTCP) = 3
+0.0    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0.0    fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
+0.0    fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0

// Start a connection

+0.0    connect(3, ..., ...) = -1  EINPROGRESS (Operation now in progress)
+0.0      >  S   0:0(0)                    <mss 1460, sackOK, TS val 100 ecr 0,   nop, wscale 8, mpcapable v1 flags[flag_h] nokey>

// set back to block. With the above setting TCP will timeout after 3 secs
+0.0        fcntl(3, F_SETFL, O_RDWR) = 0
+3...3.1  connect(3, ..., ...) = -1 ETIMEDOUT (Connection timed out)

`/tmp/sysctl_restore_${PPID}.sh`

Basically tcp-level close with timeout are ignored at mptcp level.

I have a patch for this latter problem which is a little invasive. I'll try to share soon a simpler form

pabeni commented 1 year ago

I have a patch for this latter problem which is a little invasive. I'll try to share soon a simpler form

It turned out that the simplified patch I was hoping for has negative side effects. We should resort to the full solution I just posted on the ML. (same link as #431).

daire-byrne commented 1 year ago

Yep, got it and applied - thanks! I had to munge it ever so slightly to get it to apply cleanly to v6.4.11 but I think I got it right.

I tested it with the "artificial" steps described above (down an endpoint interface and bring it back up again), but the same thing happens - the tcp connections close, the rsyncd process on serverB exits, but the client rsync on serverA hangs indefinitely (30 minutes and counting...).

I will continue to monitor it with our production workloads and see if it has had any effect on the SYN-SENT hangs (#431).

pabeni commented 1 year ago

I tested it with the "artificial" steps described above (down an endpoint interface and bring it back up again), but the same thing happens - the tcp connections close, the rsyncd process on serverB exits, but the client rsync on serverA hangs indefinitely (30 minutes and counting...).

@daire-byrne: I can't reproduce such behavior here. Could you please report the relevant part of the ss -tMneim output and nstat after the rsync transfer start, just before and just after you turn down the ens256 device, and finally just before and just after you turn it up again?

daire-byrne commented 1 year ago

After rsync start (transfer working well and using both ens224 & ens256):

serverA # ss -neimMtp | grep rsync -A1; nstat
tcp   ESTAB 0      0       10.13.21.251:50609 10.17.21.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1001 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t2,tb46080,f0,w0,o0,bl0,d41) ts sack cubic wscale:14,14 rto:245 rtt:44.692/6.954 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:20 bytes_sent:335271 bytes_acked:335272 bytes_received:1159125498 segs_out:448337 segs_in:862789 data_segs_out:562 data_segs_in:862673 send 4961962bps lastsnd:3138 lastrcv:4 lastack:4 pacing_rate 9923920bps delivery_rate 3053872bps delivered:563 app_limited busy:17309ms rcv_rtt:41.96 rcv_space:14480 rcv_ssthresh:103943250 minrtt:36.134 rcv_ooopack:7891 snd_wnd:294912 tcp-ulp-mptcp flags:Jjecv token:ed87711d(id:2)/79d85c47(id:2) seq:36530133c0c97c89 sfseq:45167531 ssnoff:bdebc6d8 maplen:68d8                                  
tcp   ESTAB 0      0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1002 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb59904,f0,w0,o0,bl0,d0) ts sack cubic wscale:14,14 rto:237 rtt:36.548/0.552 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:42 bytes_sent:65600 bytes_acked:65601 bytes_received:87359 segs_out:106 segs_in:123 data_segs_out:67 data_segs_in:74 send 12742038bps lastsnd:44958 lastrcv:49866 lastack:44922 pacing_rate 25483896bps delivery_rate 3642728bps delivered:68 app_limited busy:895ms rcv_rtt:37.504 rcv_space:14480 rcv_ssthresh:167984 minrtt:36.126 snd_wnd:131072 tcp-ulp-mptcp flags:MmBbec token:0000(id:0)/79d85c47(id:0) seq:36530133346617b9 sfseq:d568 ssnoff:c82f208f maplen:7fd8                                                                       
--
tcp   ESTAB 0      627     10.13.22.251:57963 10.17.22.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) timer:(on,240ms,0) ino:44265 sk:1003 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb46080,f2573,w1523,o0,bl0,d188) ts sack cubic wscale:14,14 rto:245 rtt:44.077/6.728 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:10 bytes_sent:187520 bytes_acked:186894 bytes_received:1196229712 segs_out:461541 segs_in:889887 data_segs_out:279 data_segs_in:889833 send 2515598bps lastsnd:5 lastrcv:4 lastack:4 pacing_rate 5031120bps delivery_rate 1207496bps delivered:279 app_limited busy:8304ms unacked:1 rcv_rtt:45.216 rcv_space:14480 rcv_ssthresh:103943250 minrtt:36.165 rcv_ooopack:10929 snd_wnd:294912 tcp-ulp-mptcp flags:Jjecv token:ed87711d(id:3)/79d85c47(id:3) seq:36530133c0c9e561 sfseq:474c6fc9 ssnoff:add8c3c5 maplen:9690
--
mptcp ESTAB 54272  0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1004 cgroup:unreachable:1 <->
     skmem:(r54272,rb415773000,t0,tb59904,f3162,w934,o0,bl0,d0) subflows:2 add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:79d85c47 write_seq:6eea3ea3549ad0da snd_una:6eea3ea3549ace67 rcv_nxt:36530133c0c9e153                                                                                                                                                                                                                                                                                                                                                                                                                                
#kernel
IpInReceives                    1409771            0.0
IpInDelivers                    1409771            0.0
IpOutRequests                   720917             0.0
IcmpInMsgs                      483                0.0
IcmpInEchos                     225                0.0
IcmpInEchoReps                  258                0.0
IcmpOutMsgs                     485                0.0
IcmpOutEchos                    260                0.0
IcmpOutEchoReps                 225                0.0
IcmpMsgInType0                  258                0.0
IcmpMsgInType8                  225                0.0
IcmpMsgOutType0                 225                0.0
IcmpMsgOutType8                 260                0.0
TcpActiveOpens                  9                  0.0
TcpPassiveOpens                 1                  0.0
TcpEstabResets                  8                  0.0
TcpInSegs                       1409287            0.0
TcpOutSegs                      720637             0.0
TcpOutRsts                      8                  0.0
UdpInDatagrams                  1                  0.0
UdpOutDatagrams                 14                 0.0
TcpExtTW                        2                  0.0
TcpExtDelayedACKs               19                 0.0
TcpExtDelayedACKLocked          1                  0.0
TcpExtDelayedACKLost            44                 0.0
TcpExtTCPHPHits                 73                 0.0
TcpExtTCPPureAcks               360                0.0
TcpExtTCPHPAcks                 13390              0.0
TcpExtTCPBacklogCoalesce        252                0.0
TcpExtTCPDSACKOldSent           44                 0.0
TcpExtTCPAbortOnData            8                  0.0
TcpExtTCPRcvCoalesce            4970               0.0
TcpExtTCPOFOQueue               5350               0.0
TcpExtTCPAutoCorking            314                0.0
TcpExtTCPOrigDataSent           24561              0.0
TcpExtTCPKeepAlive              6                  0.0
TcpExtTCPDelivered              24398              0.0
TcpExtTCPAckCompressed          5144               0.0
IpExtInOctets                   1988064218         0.0
IpExtOutOctets                  49288855           0.0
IpExtInNoECTPkts                1409770            0.0
MPTcpExtMPTCPRetrans            1                  0.0
MPTcpExtOFOQueueTail            717153             0.0
MPTcpExtOFOQueue                731678             0.0
MPTcpExtOFOMerge                681907             0.0
MPTcpExtRcvWndShared            15                 0.0
MPTcpExtRcvWndConflict          56                 0.0

serverB # ss -neimMtp | grep rsync -A1; nstat
tcp   ESTAB 0      98139   10.17.22.251:873   10.13.22.251:57963 users:(("rsync",pid=2470,fd=6)) timer:(on,083ms,0) ino:39690 sk:1 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb14575104,f677,w101723,o0,bl0,d0) ts sack cubic wscale:14,14 rto:242 rtt:41.175/9.053 ato:53 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:1175 ssthresh:929 bytes_sent:1227158533 bytes_retrans:2690272 bytes_acked:1224371220 bytes_received:200602 segs_out:912840 segs_in:472093 data_segs_out:912784 data_segs_in:298 send 316415301bps lastsnd:2 lastrcv:2695 lastack:53 pacing_rate 379691440bps delivery_rate 307675816bps delivered:910918 busy:50960ms unacked:72 retrans:0/1983 dsack_dups:188 rcv_rtt:43.25 rcv_space:14600 rcv_ssthresh:283656 notsent:1098 minrtt:36 snd_wnd:103956480 tcp-ulp-mptcp flags:Jec token:0000(id:3)/ed87711d(id:3) seq:6eea3ea3549afe60 sfseq:30a07 ssnoff:8ca39490 maplen:594                         
--
tcp   ESTAB 0      13430   10.17.21.251:873   10.13.21.251:50609 users:(("rsync",pid=2470,fd=6)) timer:(on,084ms,0) ino:39690 sk:4 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb14054400,f3466,w17014,o0,bl0,d0) ts sack cubic wscale:14,14 rto:243 rtt:42.021/10.154 ato:56 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:1627 ssthresh:1324 bytes_sent:1200299225 bytes_retrans:2200504 bytes_acked:1198086291 bytes_received:380971 segs_out:893470 segs_in:462914 data_segs_out:893345 data_segs_in:633 send 429313343bps lastsnd:3 lastrcv:120 lastack:7 pacing_rate 515169880bps delivery_rate 433274960bps delivered:891755 busy:51127ms sndbuf_limited:6ms(0.0%) unacked:10 retrans:0/1622 dsack_dups:41 rcv_rtt:45.436 rcv_space:14600 rcv_ssthresh:283656 notsent:1000 minrtt:36 snd_wnd:103956480 tcp-ulp-mptcp flags:Jec token:0000(id:2)/ed87711d(id:2) seq:6eea3ea3549bb12f sfseq:5cae3 ssnoff:aab9cea7 maplen:549
--
tcp   ESTAB 0      0       10.17.20.251:873   10.13.20.251:35402 users:(("rsync",pid=2470,fd=6)) timer:(keepalive,119min,0) ino:39690 sk:6 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb239616,f0,w0,o0,bl0,d0) ts sack cubic wscale:14,14 rto:242 rtt:41.436/7.917 ato:75 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:52 bytes_sent:87359 bytes_acked:87359 bytes_received:65600 segs_out:122 segs_in:106 data_segs_out:74 data_segs_in:67 send 13914857bps lastsnd:53084 lastrcv:48138 lastack:48138 pacing_rate 27829288bps delivery_rate 5813344bps delivered:75 app_limited busy:470ms sndbuf_limited:31ms(6.6%) rcv_rtt:42.563 rcv_space:14600 rcv_ssthresh:121636 minrtt:36.122 snd_wnd:103956480 tcp-ulp-mptcp flags:MBbec token:0000(id:0)/ed87711d(id:0) seq:6eea3ea35492d879 sfseq:fd1c ssnoff:16d39369 maplen:325                                                                                                 
--
mptcp ESTAB 0      0       10.17.20.251:873   10.13.20.251:35402 users:(("rsync",pid=2470,fd=6)) ino:39690 sk:1001 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb14575104,f2872,w111816,o0,bl0,d0) subflows:2 add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:ed87711d write_seq:36530133c4cc1009 snd_una:36530133c4ca5c38 rcv_nxt:6eea3ea3549bb678                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#kernel
IpInReceives                    394821             0.0
IpInDelivers                    394821             0.0
IpOutRequests                   43310              0.0
IcmpInMsgs                      380                0.0
IcmpInEchos                     208                0.0
IcmpInEchoReps                  172                0.0
IcmpOutMsgs                     381                0.0
IcmpOutEchos                    173                0.0
IcmpOutEchoReps                 208                0.0
IcmpMsgInType0                  172                0.0
IcmpMsgInType8                  208                0.0
IcmpMsgOutType0                 208                0.0
IcmpMsgOutType8                 173                0.0
TcpActiveOpens                  11                 0.0
TcpPassiveOpens                 2                  0.0
TcpAttemptFails                 2                  0.0
TcpEstabResets                  8                  0.0
TcpInSegs                       394441             0.0
TcpOutSegs                      792037             0.0
TcpRetransSegs                  292                0.0
TcpOutRsts                      10                 0.0
UdpOutDatagrams                 15                 0.0
Ip6InReceives                   5                  0.0
Ip6InDelivers                   5                  0.0
Ip6OutRequests                  6                  0.0
Ip6InMcastPkts                  3                  0.0
Ip6OutMcastPkts                 4                  0.0
Ip6InOctets                     368                0.0
Ip6OutOctets                    424                0.0
Ip6InMcastOctets                228                0.0
Ip6OutMcastOctets               284                0.0
Ip6InNoECTPkts                  5                  0.0
Icmp6InMsgs                     3                  0.0
Icmp6OutMsgs                    4                  0.0
Icmp6InGroupMembQueries         3                  0.0
Icmp6OutRouterSolicits          1                  0.0
Icmp6OutMLDv2Reports            3                  0.0
Icmp6InType130                  3                  0.0
Icmp6OutType133                 1                  0.0
Icmp6OutType143                 3                  0.0
TcpExtTW                        3                  0.0
TcpExtDelayedACKs               17                 0.0
TcpExtTCPHPHits                 38                 0.0
TcpExtTCPPureAcks               393465             0.0
TcpExtTCPHPAcks                 49                 0.0
TcpExtTCPSackRecovery           5                  0.0
TcpExtTCPFastRetrans            292                0.0
TcpExtTCPBacklogCoalesce        242                0.0
TcpExtTCPDSACKRecv              44                 0.0
TcpExtTCPAbortOnData            8                  0.0
TcpExtTCPSackShifted            44                 0.0
TcpExtTCPSackMerged             130                0.0
TcpExtTCPSackShiftFallback      342                0.0
TcpExtTCPRcvCoalesce            3                  0.0
TcpExtTCPAutoCorking            7414               0.0
TcpExtTCPOrigDataSent           791854             0.0
TcpExtTCPKeepAlive              5                  0.0
TcpExtTCPDelivered              792073             0.0
TcpExtTCPDSACKRecvSegs          44                 0.0
IpExtInMcastPkts                2                  0.0
IpExtInOctets                   25705382           0.0
IpExtOutOctets                  1066920498         0.0
IpExtInMcastOctets              72                 0.0
IpExtInNoECTPkts                394821             0.0
MPTcpExtOFOQueueTail            1                  0.0
MPTcpExtOFOQueue                7                  0.0
MPTcpExtDuplicateData           1                  0.0

After "ifconfig ens256 down" on serverB (rsyncd sending to client on serverA):

serverA # ss -neimMtp | grep rsync -A1; nstat
tcp   ESTAB 0      20      10.13.21.251:50609 10.17.21.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) timer:(on,218ms,0) ino:44265 sk:1001 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb46080,f2284,w1812,o0,bl0,d139) ts sack cubic wscale:14,14 rto:241 rtt:40.961/3.966 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:45 bytes_sent:1225474 bytes_acked:1225470 bytes_received:3543212079 segs_out:1330103 segs_in:2636715 data_segs_out:2047 data_segs_in:2636336 send 12181343bps lastsnd:23 lastrcv:4 lastack:4 pacing_rate 24362160bps delivery_rate 4894008bps delivered:2047 app_limited busy:66616ms unacked:1 rcv_rtt:40.126 rcv_space:14480 rcv_ssthresh:103943250 notsent:15 minrtt:36.123 rcv_ooopack:22303 snd_wnd:294912 tcp-ulp-mptcp flags:Jjecv token:ed87711d(id:2)/79d85c47(id:2) seq:36530134c1808f42 sfseq:d3304c00 ssnoff:bdebc6d8 maplen:d577
tcp   ESTAB 0      0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1002 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb59904,f0,w0,o0,bl0,d0) ts sack cubic wscale:14,14 rto:237 rtt:36.548/0.552 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:42 bytes_sent:65600 bytes_acked:65601 bytes_received:87359 segs_out:106 segs_in:123 data_segs_out:67 data_segs_in:74 send 12742038bps lastsnd:177391 lastrcv:182299 lastack:177355 pacing_rate 25483896bps delivery_rate 3642728bps delivered:68 app_limited busy:895ms rcv_rtt:37.504 rcv_space:14480 rcv_ssthresh:167984 minrtt:36.126 snd_wnd:131072 tcp-ulp-mptcp flags:MmBbec token:0000(id:0)/79d85c47(id:0) seq:36530133346617b9 sfseq:d568 ssnoff:c82f208f maplen:7fd8                                                                                          
tcp   ESTAB 0      0       10.13.22.251:57963 10.17.22.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1003 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb46080,f0,w0,o0,bl0,d319) ts sack cubic wscale:14,14 rto:242 rtt:41.401/4.119 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:31 bytes_sent:665180 bytes_acked:665181 bytes_received:3119644826 segs_out:1174454 segs_in:2324097 data_segs_out:1064 data_segs_in:2323920 send 8302408bps lastsnd:34326 lastrcv:12653 lastack:12653 pacing_rate 16604464bps delivery_rate 2992976bps delivered:1065 app_limited busy:34426ms rcv_rtt:69.229 rcv_space:14480 rcv_ssthresh:103943250 minrtt:36.163 rcv_ooopack:24083 snd_wnd:294912 tcp-ulp-mptcp flags:Jjec token:ed87711d(id:3)/79d85c47(id:3) seq:36530134b394cd18 sfseq:b9f101fb ssnoff:add8c3c5 maplen:fea0                                       
--
mptcp ESTAB 0      0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1004 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb59904,f3907,w189,o0,bl0,d0) subflows:2 add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:79d85c47 write_seq:6eea3ea354afb020 snd_una:6eea3ea354afb00c rcv_nxt:36530134c1816372                                                                                                                                                                                                                                                                                                                                                                                                                                                          
#kernel
IpInReceives                    3253267            0.0
IpInDelivers                    3253267            0.0
IpOutRequests                   1672761            0.0
IcmpInMsgs                      1581               0.0
IcmpInEchos                     804                0.0
IcmpInEchoReps                  777                0.0
IcmpOutMsgs                     1579               0.0
IcmpOutEchos                    775                0.0
IcmpOutEchoReps                 804                0.0
IcmpMsgInType0                  777                0.0
IcmpMsgInType8                  804                0.0
IcmpMsgOutType0                 804                0.0
IcmpMsgOutType8                 775                0.0
TcpActiveOpens                  40                 0.0
TcpPassiveOpens                 8                  0.0
TcpAttemptFails                 2                  0.0
TcpEstabResets                  35                 0.0
TcpInSegs                       3251657            0.0
TcpOutSegs                      1671373            0.0
TcpOutRsts                      36                 0.0
UdpInDatagrams                  27                 0.0
UdpOutDatagrams                 92                 0.0
UdpIgnoredMulti                 1                  0.0
Ip6InReceives                   5                  0.0
Ip6InDelivers                   5                  0.0
Ip6OutRequests                  8                  0.0
Ip6InMcastPkts                  3                  0.0
Ip6OutMcastPkts                 6                  0.0
Ip6InOctets                     368                0.0
Ip6OutOctets                    528                0.0
Ip6InMcastOctets                228                0.0
Ip6OutMcastOctets               388                0.0
Ip6InNoECTPkts                  5                  0.0
Icmp6InMsgs                     3                  0.0
Icmp6OutMsgs                    6                  0.0
Icmp6InGroupMembQueries         3                  0.0
Icmp6OutRouterSolicits          3                  0.0
Icmp6OutMLDv2Reports            3                  0.0
Icmp6InType130                  3                  0.0
Icmp6OutType133                 3                  0.0
Icmp6OutType143                 3                  0.0
TcpExtTW                        12                 0.0
TcpExtDelayedACKs               78                 0.0
TcpExtDelayedACKLocked          3                  0.0
TcpExtDelayedACKLost            230                0.0
TcpExtTCPHPHits                 209                0.0
TcpExtTCPPureAcks               563                0.0
TcpExtTCPHPAcks                 43034              0.0
TcpExtTCPBacklogCoalesce        735                0.0
TcpExtTCPDSACKOldSent           230                0.0
TcpExtTCPAbortOnData            34                 0.0
TcpExtTCPRcvCoalesce            25573              0.0
TcpExtTCPOFOQueue               27566              0.0
TcpExtTCPAutoCorking            631                0.0
TcpExtTCPOrigDataSent           78637              0.0
TcpExtTCPKeepAlive              20                 0.0
TcpExtTCPDelivered              78909              0.0
TcpExtTCPAckCompressed          26564              0.0
TcpExtTcpDuplicateDataRehash    4                  0.0
IpExtInMcastPkts                3                  0.0
IpExtInBcastPkts                1                  0.0
IpExtInOctets                   4566916846         0.0
IpExtOutOctets                  115667366          0.0
IpExtInMcastOctets              108                0.0
IpExtInBcastOctets              229                0.0
IpExtInNoECTPkts                3253270            0.0
MPTcpExtOFOQueueTail            1627097            0.0
MPTcpExtOFOQueue                1660449            0.0
MPTcpExtOFOMerge                1546787            0.0
MPTcpExtDuplicateData           405                0.0
MPTcpExtRcvWndShared            69                 0.0
MPTcpExtRcvWndConflictUpdate    2                  0.0
MPTcpExtRcvWndConflict          167                0.0

serverB # ss -neimMtp | grep rsync -A1; nstat
tcp   ESTAB 0      1148823  10.17.22.251:873   10.13.22.251:57963 users:(("rsync",pid=2470,fd=6)) timer:(on,5.308ms,5) ino:39690 sk:1 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb14575104,f2025,w1198103,o0,bl0,d0) ts sack cubic wscale:14,14 rto:7584 backoff:5 rtt:36.955/0.352 ato:53 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:1 ssthresh:978 bytes_sent:3123176652 bytes_retrans:3531826 bytes_acked:3118627107 bytes_received:665180 segs_out:2326382 segs_in:1174068 data_segs_out:2326206 data_segs_in:1064 send 300041bps lastsnd:2276 lastrcv:31166 lastack:9707 pacing_rate 503348088bps delivery_rate 392886584bps delivered:2323153 busy:172853ms sndbuf_limited:58ms(0.0%) unacked:763 retrans:1/2605 lost:763 dsack_dups:314 rcv_rtt:42.142 rcv_space:14600 rcv_ssthresh:283656 notsent:131104 minrtt:36 snd_wnd:103956480 tcp-ulp-mptcp flags:Jec token:0000(id:3)/ed87711d(id:3) seq:6eea3ea354ac1437 sfseq:9fc51 ssnoff:8ca39490 maplen:2a0c
--
tcp   ESTAB 0      151441   10.17.21.251:873   10.13.21.251:50609 users:(("rsync",pid=2470,fd=6)) timer:(on,063ms,0) ino:39690 sk:4 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb14054400,f2031,w157713,o0,bl0,d0) ts sack cubic wscale:14,14 rto:237 rtt:36.687/0.103 ato:105 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:2650 ssthresh:1128 bytes_sent:3522829092 bytes_retrans:3284460 bytes_acked:3519393703 bytes_received:1191312 segs_out:2621350 segs_in:1321204 data_segs_out:2620986 data_segs_in:1965 send 800915856bps lastsnd:17 lastrcv:14 lastack:13 pacing_rate 961079376bps delivery_rate 758287792bps delivered:2618591 app_limited busy:173230ms sndbuf_limited:435ms(0.3%) unacked:113 retrans:0/2422 dsack_dups:139 rcv_rtt:49.663 rcv_space:14600 rcv_ssthresh:283656 notsent:512 minrtt:36 snd_wnd:103956480 tcp-ulp-mptcp flags:Jec token:0000(id:2)/ed87711d(id:2) seq:6eea3ea354af2a9a sfseq:122d8c ssnoff:aab9cea7 maplen:5             
--
tcp   ESTAB 0      0        10.17.20.251:873   10.13.20.251:35402 users:(("rsync",pid=2470,fd=6)) timer:(keepalive,117min,0) ino:39690 sk:6 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb239616,f0,w0,o0,bl0,d0) ts sack cubic wscale:14,14 rto:242 rtt:41.436/7.917 ato:75 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:52 bytes_sent:87359 bytes_acked:87359 bytes_received:65600 segs_out:122 segs_in:106 data_segs_out:74 data_segs_in:67 send 13914857bps lastsnd:179179 lastrcv:174233 lastack:174233 pacing_rate 27829288bps delivery_rate 5813344bps delivered:75 app_limited busy:470ms sndbuf_limited:31ms(6.6%) rcv_rtt:42.563 rcv_space:14600 rcv_ssthresh:121636 minrtt:36.122 snd_wnd:103956480 tcp-ulp-mptcp flags:MBbec token:0000(id:0)/ed87711d(id:0) seq:6eea3ea35492d879 sfseq:fd1c ssnoff:16d39369 maplen:325                                                                                                                                  
--
mptcp ESTAB 896    0        10.17.20.251:873   10.13.20.251:35402 users:(("rsync",pid=2470,fd=6)) ino:39690 sk:1001 cgroup:unreachable:1 <->
     skmem:(r896,rb1136800,t0,tb14575104,f3878,w151770,o0,bl0,d0) subflows:2 add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:ed87711d write_seq:36530134c018427b snd_una:36530134c015f2ea rcv_nxt:6eea3ea354af2a9f                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#kernel
IpInReceives                    1562544            0.0
IpInDelivers                    1562544            0.0
IpOutRequests                   170696             0.0
IcmpInMsgs                      1512               0.0
IcmpInEchos                     750                0.0
IcmpInEchoReps                  762                0.0
IcmpOutMsgs                     1513               0.0
IcmpOutEchos                    763                0.0
IcmpOutEchoReps                 750                0.0
IcmpMsgInType0                  762                0.0
IcmpMsgInType8                  750                0.0
IcmpMsgOutType0                 750                0.0
IcmpMsgOutType8                 763                0.0
TcpActiveOpens                  45                 0.0
TcpPassiveOpens                 11                 0.0
TcpAttemptFails                 4                  0.0
TcpEstabResets                  38                 0.0
TcpInSegs                       1561012            0.0
TcpOutSegs                      3140973            0.0
TcpRetransSegs                  1423               0.0
TcpOutRsts                      41                 0.0
UdpInDatagrams                  22                 0.0
UdpOutDatagrams                 86                 0.0
Ip6InReceives                   6                  0.0
Ip6InDelivers                   6                  0.0
Ip6OutRequests                  8                  0.0
Ip6InMcastPkts                  2                  0.0
Ip6OutMcastPkts                 4                  0.0
Ip6InOctets                     432                0.0
Ip6OutOctets                    536                0.0
Ip6InMcastOctets                152                0.0
Ip6OutMcastOctets               256                0.0
Ip6InNoECTPkts                  6                  0.0
Icmp6InMsgs                     2                  0.0
Icmp6OutMsgs                    4                  0.0
Icmp6InGroupMembQueries         2                  0.0
Icmp6OutRouterSolicits          2                  0.0
Icmp6OutMLDv2Reports            2                  0.0
Icmp6InType130                  2                  0.0
Icmp6OutType133                 2                  0.0
Icmp6OutType143                 2                  0.0
TcpExtTW                        15                 0.0
TcpExtDelayedACKs               59                 0.0
TcpExtDelayedACKLocked          1                  0.0
TcpExtTCPHPHits                 162                0.0
TcpExtTCPPureAcks               1557665            0.0
TcpExtTCPHPAcks                 260                0.0
TcpExtTCPSackRecovery           18                 0.0
TcpExtTCPLostRetransmit         4                  0.0
TcpExtTCPFastRetrans            1417               0.0
TcpExtTCPTimeouts               5                  0.0
TcpExtTCPLossProbes             1                  0.0
TcpExtTCPBacklogCoalesce        717                0.0
TcpExtTCPDSACKRecv              224                0.0
TcpExtTCPAbortOnData            37                 0.0
TcpExtTCPSackShifted            201                0.0
TcpExtTCPSackMerged             738                0.0
TcpExtTCPSackShiftFallback      1524               0.0
TcpExtTCPRcvCoalesce            18                 0.0
TcpExtTCPAutoCorking            29767              0.0
TcpExtTCPSynRetrans             1                  0.0
TcpExtTCPOrigDataSent           3140255            0.0
TcpExtTCPKeepAlive              20                 0.0
TcpExtTCPDelivered              3139654            0.0
TcpExtTcpTimeoutRehash          5                  0.0
TcpExtTCPDSACKRecvSegs          224                0.0
IpExtInMcastPkts                2                  0.0
IpExtInOctets                   101427176          0.0
IpExtOutOctets                  4232626083         0.0
IpExtInMcastOctets              72                 0.0
IpExtInNoECTPkts                1562548            0.0
MPTcpExtMPTCPRetrans            5                  0.0
MPTcpExtOFOQueue                4                  0.0
MPTcpExtSubflowStale            1                  0.0
MPTcpExtSndWndShared            1                  0.0

After bringing back up ens256 on serverB (but without correct route to ens256 on serverA) - the client is now hung on serverA:

serverA # ss -neimMtp | grep rsync -A1; nstat
tcp   CLOSE-WAIT 1      0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1002 cgroup:unreachable:1 -->
     skmem:(r0,rb415773000,t0,tb59904,f0,w0,o0,bl0,d0) ts sack cubic wscale:14,14 rto:237 rtt:36.548/0.552 ato:40 mss:1386 pmtu:1500 rcvmss:1358 advmss:1448 cwnd:42 bytes_sent:65600 bytes_acked:65601 bytes_received:87360 segs_out:107 segs_in:126 data_segs_out:67 data_segs_in:74 send 12742038bps lastsnd:305762 lastrcv:310670 lastack:19106 pacing_rate 25483896bps delivery_rate 3642728bps delivered:68 app_limited busy:895ms rcv_rtt:37.504 rcv_space:14480 rcv_ssthresh:167984 minrtt:36.126 snd_wnd:294912 tcp-ulp-mptcp flags:MmBbec token:0000(id:0)/79d85c47(id:0) seq:36530133346617b9 sfseq:d568 ssnoff:c82f208f maplen:7fd8
--
mptcp ESTAB      0      0       10.13.20.251:35402 10.17.20.251:873   users:(("rsync",pid=2570,fd=3),("rsync",pid=2569,fd=3)) ino:44265 sk:1004 cgroup:unreachable:1 <->
     skmem:(r0,rb415773000,t0,tb59904,f0,w0,o0,bl0,d0) add_addr_signal:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:79d85c47 write_seq:6eea3ea354c84105 snd_una:6eea3ea354c84105 rcv_nxt:365301351b8ee4fb                                                                                                                                                                                                                                                                                                                                                                                                   
#kernel
IpInReceives                    141                0.0
IpInDelivers                    141                0.0
IpOutRequests                   151                0.0
IcmpInMsgs                      75                 0.0
IcmpInEchos                     42                 0.0
IcmpInEchoReps                  33                 0.0
IcmpOutMsgs                     75                 0.0
IcmpOutEchos                    33                 0.0
IcmpOutEchoReps                 42                 0.0
IcmpMsgInType0                  33                 0.0
IcmpMsgInType8                  42                 0.0
IcmpMsgOutType0                 42                 0.0
IcmpMsgOutType8                 33                 0.0
TcpActiveOpens                  3                  0.0
TcpPassiveOpens                 1                  0.0
TcpEstabResets                  3                  0.0
TcpInSegs                       66                 0.0
TcpOutSegs                      78                 0.0
TcpOutRsts                      3                  0.0
Ip6OutRequests                  1                  0.0
Ip6OutMcastPkts                 1                  0.0
Ip6OutOctets                    56                 0.0
Ip6OutMcastOctets               56                 0.0
Icmp6OutMsgs                    1                  0.0
Icmp6OutRouterSolicits          1                  0.0
Icmp6OutType133                 1                  0.0
TcpExtTW                        1                  0.0
TcpExtDelayedACKs               3                  0.0
TcpExtTCPHPHits                 12                 0.0
TcpExtTCPPureAcks               18                 0.0
TcpExtTCPHPAcks                 26                 0.0
TcpExtTCPAbortOnData            3                  0.0
TcpExtTCPRcvCoalesce            2                  0.0
TcpExtTCPAutoCorking            4                  0.0
TcpExtTCPOrigDataSent           48                 0.0
TcpExtTCPKeepAlive              2                  0.0
TcpExtTCPDelivered              47                 0.0
IpExtInOctets                   14169              0.0
IpExtOutOctets                  21844              0.0
IpExtInNoECTPkts                141                0.0

serverB # ss -neimMtp | grep rsync -A1; nstat
mptcp ESTAB 0      0       10.17.20.251:873   10.13.20.251:35402 users:(("rsync",pid=2470,fd=6)) ino:39690 sk:1001 cgroup:unreachable:1 <->
     skmem:(r0,rb1136800,t0,tb19137024,f3586,w12788222,o0,bl0,d0) add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_accepted_max:4 remote_key token:ed87711d write_seq:365301351c51c9e1 snd_una:365301351b8ee4fb rcv_nxt:6eea3ea354c84105                                                                                                                                                                                                                                                                                              
#kernel
IpInReceives                    1116               0.0
IpInDelivers                    1116               0.0
IpOutRequests                   1188               0.0
IcmpInMsgs                      805                0.0
IcmpInEchos                     399                0.0
IcmpInEchoReps                  406                0.0
IcmpOutMsgs                     805                0.0
IcmpOutEchos                    406                0.0
IcmpOutEchoReps                 399                0.0
IcmpMsgInType0                  406                0.0
IcmpMsgInType8                  399                0.0
IcmpMsgOutType0                 399                0.0
IcmpMsgOutType8                 406                0.0
TcpActiveOpens                  21                 0.0
TcpPassiveOpens                 6                  0.0
TcpEstabResets                  20                 0.0
TcpInSegs                       306                0.0
TcpOutSegs                      375                0.0
TcpOutRsts                      20                 0.0
UdpInDatagrams                  3                  0.0
UdpOutDatagrams                 23                 0.0
Ip6InReceives                   3                  0.0
Ip6InDelivers                   3                  0.0
Ip6OutRequests                  6                  0.0
Ip6InMcastPkts                  3                  0.0
Ip6OutMcastPkts                 6                  0.0
Ip6InOctets                     228                0.0
Ip6OutOctets                    396                0.0
Ip6InMcastOctets                228                0.0
Ip6OutMcastOctets               396                0.0
Ip6InNoECTPkts                  3                  0.0
Icmp6InMsgs                     3                  0.0
Icmp6OutMsgs                    6                  0.0
Icmp6InGroupMembQueries         3                  0.0
Icmp6OutRouterSolicits          3                  0.0
Icmp6OutMLDv2Reports            3                  0.0
Icmp6InType130                  3                  0.0
Icmp6OutType133                 3                  0.0
Icmp6OutType143                 3                  0.0
TcpExtTW                        7                  0.0
TcpExtDelayedACKs               17                 0.0
TcpExtTCPHPHits                 63                 0.0
TcpExtTCPPureAcks               67                 0.0
TcpExtTCPHPAcks                 123                0.0
TcpExtTCPAbortOnData            20                 0.0
TcpExtTCPRcvCoalesce            12                 0.0
TcpExtTCPAutoCorking            5                  0.0
TcpExtTCPOrigDataSent           203                0.0
TcpExtTCPKeepAlive              11                 0.0
TcpExtTCPDelivered              222                0.0
IpExtInMcastPkts                2                  0.0
IpExtInOctets                   108622             0.0
IpExtOutOctets                  199857             0.0
IpExtInMcastOctets              72                 0.0
IpExtInNoECTPkts                1116               0.0

So this time I can see a tcp connection in CLOSE-WAIT on serverA which I don't think I saw before (both servers just had the "MPTCP ESTAB" entries with no TCP connections).

Both the client rsync process (serverA) and the rsyncd process (serverB) hang indefinitely in this state.

If I kill the client rsync process on serverA, the corresponding rsyncd process on serverB continues to hang indefinitely.

I hope that helps.

I am still waiting to see any actual hangs with our production workloads and I am less concerned about the hangs caused by the steps in the ticket as I am obviously breaking the mptcp connections in an avoidable way.

pabeni commented 1 year ago

If I kill the client rsync process on serverA, the corresponding rsyncd process on serverB continues to hang indefinitely.

I hope that helps.

Thanks, indeed it does! There is (at least) a bug in the shared patches. Specifically the close timeout will never fire - or to be more accururate, it will fire in ~(LONG_MAX/(2*HZ) - ) seconds.

I'll try to shared a fixed version of the patch ASAP.

pabeni commented 1 year ago

I'll try to shared a fixed version of the patch ASAP.

this version should address the bug mentioned above:

https://patchwork.kernel.org/project/mptcp/list/?series=780262

the 2 patches should replace the older ones. @daire-byrne: it would be great if you could spin them in your testbed.

Side note: it's unclear to me why the TCP subflows close after the device goes up. I guess the one on top of flipping device start using the default route and a different IP-level path. The peer could end-up resetting the connection due to rp_filter and/or f/w. But the other subflows should stay alive...

daire-byrne commented 1 year ago

New v2 patch applied but I still see the same behaviour when the interface on serverB is brought back online:

serverA # [root@chdtransfer1 ~]# ss -neimMtp | grep rsync -A1; nstat
mptcp ESTAB 0      0       10.13.20.251:47424 10.17.20.251:873   users:(("rsync",pid=2498,fd=3),("rsync",pid=2497,fd=3)) timer:(keepalive,31sec,0) ino:44197 sk:1001 cgroup:unreachable:1 <->
     skmem:(r0,rb283724400,t0,tb59904,f3884,w212,o0,bl0,d0) add_addr_signal:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:12c9d786 write_seq:ccbc0874ace81e9a snd_una:ccbc0874ace81e1c rcv_nxt:7f0f772cff0ef8ee                                                                                                                                                                                                                                                                     
#kernel
IpInReceives                    314                0.0
IpInDelivers                    314                0.0
IpOutRequests                   333                0.0
IcmpInMsgs                      209                0.0
IcmpInEchos                     116                0.0
IcmpInEchoReps                  93                 0.0
IcmpOutMsgs                     208                0.0
IcmpOutEchos                    92                 0.0
IcmpOutEchoReps                 116                0.0
IcmpMsgInType0                  93                 0.0
IcmpMsgInType8                  116                0.0
IcmpMsgOutType0                 116                0.0
IcmpMsgOutType8                 92                 0.0
TcpActiveOpens                  5                  0.0
TcpPassiveOpens                 2                  0.0
TcpEstabResets                  5                  0.0
TcpInSegs                       105                0.0
TcpOutSegs                      137                0.0
TcpOutRsts                      5                  0.0
UdpOutDatagrams                 6                  0.0
TcpExtTW                        2                  0.0
TcpExtDelayedACKs               4                  0.0
TcpExtTCPHPHits                 9                  0.0
TcpExtTCPPureAcks               33                 0.0
TcpExtTCPHPAcks                 44                 0.0
TcpExtTCPAbortOnData            5                  0.0
TcpExtTCPRcvCoalesce            3                  0.0
TcpExtTCPAutoCorking            5                  0.0
TcpExtTCPOrigDataSent           94                 0.0
TcpExtTCPKeepAlive              5                  0.0
TcpExtTCPDelivered              98                 0.0
IpExtInOctets                   27737              0.0
IpExtOutOctets                  98095              0.0
IpExtInNoECTPkts                314                0.0

serverB # ss -neimMtp | grep rsync -A1; nstat
mptcp ESTAB    0      0       10.17.20.251:873   10.13.20.251:47424 users:(("rsync",pid=6408,fd=6)) ino:51485 sk:d cgroup:unreachable:1 <->
     skmem:(r0,rb1038200,t0,tb33467904,f1042,w22334446,o0,bl0,d0) add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_accepted_max:4 remote_key token:1c57ba07 write_seq:7f0f772d006320f4 snd_una:7f0f772cff0ef8ee rcv_nxt:ccbc0874ace81e1c                                                                                                                                                                                                                                                                                                                                                                        
#kernel
IpInReceives                    538                0.0
IpInDelivers                    538                0.0
IpOutRequests                   597                0.0
IcmpInMsgs                      374                0.0
IcmpInEchos                     197                0.0
IcmpInEchoReps                  177                0.0
IcmpOutMsgs                     374                0.0
IcmpOutEchos                    177                0.0
IcmpOutEchoReps                 197                0.0
IcmpMsgInType0                  177                0.0
IcmpMsgInType8                  197                0.0
IcmpMsgOutType0                 197                0.0
IcmpMsgOutType8                 177                0.0
TcpActiveOpens                  9                  0.0
TcpPassiveOpens                 2                  0.0
TcpEstabResets                  6                  0.0
TcpInSegs                       148                0.0
TcpOutSegs                      185                0.0
TcpOutRsts                      6                  0.0
UdpInDatagrams                  16                 0.0
UdpOutDatagrams                 42                 0.0
TcpExtTW                        4                  0.0
TcpExtDelayedACKs               11                 0.0
TcpExtTCPHPHits                 49                 0.0
TcpExtTCPPureAcks               30                 0.0
TcpExtTCPHPAcks                 68                 0.0
TcpExtTCPAbortOnData            6                  0.0
TcpExtTCPRcvCoalesce            2                  0.0
TcpExtTCPAutoCorking            7                  0.0
TcpExtTCPOrigDataSent           106                0.0
TcpExtTCPKeepAlive              4                  0.0
TcpExtTCPDelivered              105                0.0
IpExtInOctets                   56376              0.0
IpExtOutOctets                  75779              0.0
IpExtInNoECTPkts                538                0.0

Both the client and rsyncd processes hang indefinitely. I didn't catch it in CLOSE-WAIT this time...

Because I had to munge one hunk of the patch to apply it to v6.4.11, let me just include my edited hunk in case I have mistranslated it (I have some doubts):

    if (test_and_clear_bit(MPTCP_WORK_CLOSE_SUBFLOW, &msk->flags))
        __mptcp_close_subflow(sk);

-   /* There is no point in keeping around an orphaned sk timedout or
-    * closed, but we need the msk around to reply to incoming DATA_FIN,
-    * even if it is orphaned and in FIN_WAIT2 state
-    */
-   if (sock_flag(sk, SOCK_DEAD)) {
-       if (mptcp_should_close(sk)) {
-           inet_sk_state_store(sk, TCP_CLOSE);
-           mptcp_do_fastclose(sk);
-       }
-       if (sk->sk_state == TCP_CLOSE) {
-           __mptcp_destroy_sock(sk);
-           goto unlock;
-       }
+   if (mptcp_close_tout_expired(sk)) {
+       mptcp_do_fastclose(sk);
+       mptcp_close_wake_up(sk);
+   }
+
+   if (sock_flag(sk, SOCK_DEAD) && sk->sk_state == TCP_CLOSE) {
+       __mptcp_destroy_sock(sk);
+       goto unlock;
    }

    if (test_and_clear_bit(MPTCP_WORK_RTX, &msk->flags))

So we also managed to reproduce this type of hang a different way today on the production systems (also running the updated v2 patch). We had to do a firewall failover to upgrade firmware which will have caused a momentary TCP disconnect between all transfer server interfaces (ens192, ens224, ens256) between two servers pairs.

All the client rsync processes in this case were hung indefinitely but the rsyncd processes seem to have died off themselves.

mptcp ESTAB 402048  0        10.25.20.251:51628 10.29.20.251:873   users:(("rsync",pid=688488,fd=4),("rsync",pid=688479,fd=4)) timer:(keepalive,38sec,0) ino:5356652 sk:ad cgroup:unreachable:1 <->
     skmem:(r402048,rb536870912,t0,tb14155776,f3940,w156,o0,bl0,d0) add_addr_signal:2 add_addr_accepted:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:88548ac5 write_seq:107dc9c23ed19837 snd_una:107dc9c23ed1981b rcv_nxt:5323cd2ec870f857

So no playing with routes or upping and downing the server interfaces - just all active TCP connections being dropped by the firewall in between them.

pabeni commented 1 year ago

@daire-byrne: thank you for the quick feedback, even if the results are not the one I was looking for...

Can you please confirm that:

Note: we could make such timeout configurable e.g. systctl if is just a matter of too long delay - but I think there is other problem there: I expect that even in your setup the mptcp sockets will turn to close after ~60" but rsync process will not be woken-up.

So we also managed to reproduce this type of hang a different way today on the production systems (also running the updated v2 patch). We had to do a firewall failover to upgrade firmware which will have caused a momentary TCP disconnect between all transfer server interfaces (ens192, ens224, ens256) between two servers pairs.

This is consistent with my current understanding of this issue .

pabeni commented 1 year ago

@daire-byrne: addendum, could you please additionally check which syscall is blocking the relevant rsync process?

strace -f -p <rsync pid>

should tell

daire-byrne commented 1 year ago

Yea, the client rsync is definitely still hung from this morning's "artificial" test - 6 hours ago.

However, it does look like the rsyncd process on serverB eventually died due to a configured rsync timeout:

2023/08/30 15:12:57 [6408] [sender] io timeout after 901 seconds -- exiting
2023/08/30 15:12:57 [6408] sent 2009193968 bytes  received 282179 bytes  total size 2046893369
2023/08/30 15:12:57 [6408] rsync error: timeout in data send/receive (code 30) at io.c(197) [sender=3.2.3]

And the firewall failover examples were noticed because they had been hung for 15+ minutes already.

As for the client rsync strace - I think rsync actually spawns two processes (generator/sender), one for receiving from the remote host and one for operating locally:

serverA # pstree -p 2497
rsync(2497)───rsync(2498)

serverA # strace -f -p 2497
strace: Process 2497 attached
select(5, [4], [], [4], {tv_sec=36, tv_usec=663747}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 1 (out [3], left {tv_sec=59, tv_usec=999997})
write(3, "\0\0\0\7", 4)                 = 4
select(5, [4], [], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)

serverA # # strace -f -p 2498
strace: Process 2498 attached
select(4, [3], [], [3], {tv_sec=54, tv_usec=785481}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}

Could it be that the communication between them is the thing that has broken because it is using mptcp too (mptcpize/LD_PRELOAD)?

rsync      2497                           root    3u     sock                0,8       0t0      44197 protocol: MPTCP
rsync      2497                           root    4u     unix 0xffff8ef2d8a40000       0t0      44199 type=STREAM
rsync      2498                           root    3u     sock                0,8       0t0      44197 protocol: MPTCP
rsync      2498                           root    5u     unix 0xffff8ef2d8a45d80       0t0      44200 type=STREAM

Or maybe I am reading that wrong....

pabeni commented 1 year ago

As for the client rsync strace - I think rsync actually spawns two processes (generator/sender), one for receiving from the remote host and one for operating locally:

serverA # pstree -p 2497
rsync(2497)───rsync(2498)

serverA # strace -f -p 2497
strace: Process 2497 attached
select(5, [4], [], [4], {tv_sec=36, tv_usec=663747}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 1 (out [3], left {tv_sec=59, tv_usec=999997})
write(3, "\0\0\0\7", 4)                 = 4
select(5, [4], [], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)

It looks like the mptcp socket is not moving at all into the TCP_CLOSE state - which is unexpected with the tentative fix patch applied. Could you please double-check that with the ss tool (ss -Meinm)? thanks!

daire-byrne commented 1 year ago

Requested output from another example:

serverA # ss -Meinm
State                                Recv-Q                                Send-Q                                                                 Local Address:Port                                                                  Peer Address:Port                                Process                                
ESTAB                                0                                     0                                                                       10.13.20.251:43526                                                                 10.17.20.251:873                                  timer:(keepalive,54sec,0) ino:44745 sk:d cgroup:unreachable:1 <->
     skmem:(r0,rb536870912,t0,tb46080,f2006,w2090,o0,bl0,d0) add_addr_signal:2 subflows_max:4 add_addr_signal_max:2 add_addr_accepted_max:4 remote_key token:bb53f763 write_seq:c8c9e67ac115d002 snd_una:c8c9e67ac115cb0c rcv_nxt:b6ce82075ce522c2
pabeni commented 1 year ago

@daire-byrne: thanks again for the feedback. I just noticed there is at least an integer overflow in the timeout logic in the proposed fix. that makes the patch ineffective for the first ~5' after the reboot - until jiffies become positive. Are you able to reproduce the hangup on a machine with the patched kernel and uptime >> 5'?

daire-byrne commented 1 year ago

I booted serverA and serverB and waited an hour before running through the steps - still the same hang (still going after 10 mins).

Did my hunk edit look okay (above) for v6.4.11? That was my only doubt about whether I had applied the patch correctly. I have double checked the build and the patches are definitely in there.

For the similar looking hangs we saw after the firewall failover, the hosts had been up for many hours by that point.

pabeni commented 1 year ago

thanks even more for the feedback.

Yes, the hunk reported above looks correct.

At this point I don't have any other ideas.

To get to the bottom of it I need to understand why the msk never transition to the TCP_CLOSE status. The 'perf' tool could help, but the usage is not trivial. You need to

1) install the perf tool itself, which is usually packaged by your distro of choice

2) install the kernel debug info and kernel sources on the target/running host. If you compile your own kernel that means placing the 'vmlinux' file in the perf search path (usually rename it as /boot/vmlinux-$(uname -r) ) and build the kernel in the same target host (or e.g. mount the filesystem where you build at the exactly same location/dir path)

3) create a few perf probes on the target host to collect significant data. a) one probe in __mptcp_close_ssk:

CLOSE_LINE=$(perf probe -L __mptcp_close_ssk| grep 'list_is_singular' | awk '{print $1}')
perf probe -a "__mptcp_close_ssk:$CLOSE_LINE sk sk->__sk_common.skc_state msk->conn_list.next msk->conn_list.prev msk->first ssk_state=msk->first->__sk_common.skc_state msk->pm.subflows"

b) 2 probes for mptcp_worker:

WORK_LINE=$(perf probe -L mptcp_worker | grep 'mptcp_close_tout_expired' | awk '{print $1}')
perf probe -a "mptcp_worker:$WORK_LINE msk"
perf probe -a "mptcp_worker:$((WORK_LINE)) msk"

4) start perf on the target host:

perf record -a -e probe:mptcp_worker_L${WORK_LINE} -e probe:mptcp_worker_L$((WORK_LINE + 1)) - e probe:__mptcp_close_ssk_L${CLOSE_LINE} &

5) reproduce the issue

6) gracefully terminate perf (ctrl-c)

7) collect the perf data: perf script and provide them back

not trivial as you can see, but possibly very useful :) the plus side is that no reboot is needed.

daire-byrne commented 1 year ago

With such concise steps, how could I fail?!

So I grabbed the perf probes on serverA starting right before the "ifconfig ens256 up" on serverB. Then I proceeded to bring up the interface which hung the in progress mptcpize rsync. I waited a few minutes and then stopped the perf probe.

 kworker/2:3+eve   242 [002]   693.453136:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/1:0-mm_  2829 [001]   693.469860: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d807bd00 first=0xffff93b3f980ec00 ssk_state=0x8 subflows=0x1
 kworker/1:0-mm_  2829 [001]   693.469863:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   693.473901: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x8 subflows=0x0
 kworker/2:3+eve   242 [002]   693.473903:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   693.507147: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0
 kworker/2:3+eve   242 [002]   693.507151:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   693.743998:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   693.951877:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   694.159890:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   694.367871:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   694.575813:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   694.783790:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   694.991786:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   695.199774:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   695.407735:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   695.615712:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   695.823691:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   696.031668:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   696.239648:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   696.447635:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   696.655605:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   696.863585:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   697.071576:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   697.279547:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   697.487536:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   697.695552:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   697.903485:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   698.111483:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   698.319456:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   698.527432:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   698.735465:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   698.943427:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   699.151380:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   699.359372:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   699.567326:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   699.775300:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   699.983314:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   700.191270:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   700.399257:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   700.607260:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   700.815297:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   701.023190:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   701.231168:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   701.439165:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   701.647234:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   701.855121:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   702.063086:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   702.271063:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   702.479082:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   702.687024:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   702.895043:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   703.102982:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   703.310968:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   703.518944:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   703.726937:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   703.934924:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   704.142887:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   704.350882:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   704.558868:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   704.766877:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   704.974815:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   705.182785:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   705.390855:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   705.598751:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   705.806743:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   706.014714:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   706.222695:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   706.430674:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   706.638675:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   706.846653:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   707.054603:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   707.262593:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   707.470571:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   707.678558:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   707.886535:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   708.094515:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   708.302536:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   708.510483:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   708.718497:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   708.926434:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   709.134410:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   709.342394:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   709.550385:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   709.758347:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   709.966335:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   710.174304:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   710.382309:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   710.590301:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   710.798261:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   711.006230:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   711.214219:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   711.422214:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   711.630689:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   711.838170:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   712.046163:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   712.254124:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   712.462104:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   712.670072:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   712.878056:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   713.086044:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   713.294041:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   713.501993:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   713.709979:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   713.918008:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   714.125932:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   714.333928:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   714.541909:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   714.749932:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   714.957850:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   715.165848:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   715.373814:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   715.581795:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   715.789771:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   715.997805:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   716.205729:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   716.413720:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   716.621710:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   716.829690:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   717.037655:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   717.245649:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   717.453645:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   717.661598:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   717.869572:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   718.077570:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   718.285553:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   718.493587:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   718.701492:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   718.909490:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   719.117454:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   719.325443:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   719.533408:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   719.741439:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   719.949374:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   720.157350:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   720.365337:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   720.573313:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   720.781303:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   720.989289:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   721.197300:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   721.405237:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   721.613247:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   721.821195:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   722.029187:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   722.237146:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   722.445140:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   722.653127:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   722.861092:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   723.069073:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   723.277078:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   723.485063:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   723.693054:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   723.901020:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   724.108991:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   724.316963:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   724.524937:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   724.732967:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   724.940899:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   725.148886:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   725.356864:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   725.564833:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   725.772814:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   725.980799:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   726.188783:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   726.396757:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   726.604737:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   726.812762:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   727.020708:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   727.228676:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   727.436656:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   727.644630:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   727.852627:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   728.060589:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   728.268576:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   728.476550:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   728.684546:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   728.892529:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   729.100505:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   729.308475:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   729.516450:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   729.724441:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   729.932441:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   730.140418:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   730.348380:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   730.556376:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   730.764340:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   730.978073:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   731.180319:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   731.388307:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   731.596264:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   731.804247:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   732.012231:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   732.220219:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   732.428213:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   732.636156:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:1-cgr  3623 [002]   732.844147:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   733.052135:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   733.260128:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   733.468076:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   733.676103:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   733.884037:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   734.092023:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   734.300016:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   734.508028:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   734.715978:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   734.923933:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   735.131916:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   735.339909:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   735.547884:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   735.755878:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   735.963848:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   736.171824:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   736.379814:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   736.587786:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   736.795770:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   737.003759:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   737.211728:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   737.419734:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   737.627701:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   737.835679:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   738.043695:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   738.251629:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   738.459601:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   738.667579:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   738.875563:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   739.083561:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   739.291537:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   739.499521:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   739.707520:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   739.915475:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   740.123465:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   740.331453:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   740.539404:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   740.747404:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   740.955374:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   741.163347:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   741.371335:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   741.579335:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   741.787309:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   741.995285:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   742.203269:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   742.411221:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   742.619216:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   742.827211:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   743.035169:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   743.243199:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   743.451156:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   743.659105:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   743.875362:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   744.083088:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   744.291069:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   744.499052:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   744.707026:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   744.914984:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   745.122963:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   745.330960:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   745.538923:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   745.746937:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   745.954892:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   746.162857:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   746.370865:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   746.578838:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   746.786827:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   746.994791:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   747.202782:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   747.410752:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   747.618737:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   747.826714:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   748.034695:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   748.242659:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   748.450656:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   748.658633:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   748.866626:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   749.074595:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   749.282589:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   749.490837:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   749.698528:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   749.906513:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   750.114494:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   750.322497:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   750.530470:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   750.738434:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   750.946415:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   751.154398:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   751.362378:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   751.570355:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   751.778340:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   751.986311:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   752.194284:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   752.402268:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   752.610250:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   752.818338:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.026204:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.234187:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.442171:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.650160:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.650164:      probe:mptcp_worker_L24: (ffffffffaeba46a0) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   753.650171: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0
 kworker/2:3+eve   242 [002]   753.858117:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   754.066099:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   754.274081:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   754.482062:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   754.690041:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   754.898026:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   755.106008:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   755.313992:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   755.521960:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   755.729950:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   755.937939:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   756.145903:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   756.353886:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   756.561873:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   756.769863:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   756.977827:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   757.185806:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   757.393799:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   757.601794:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   757.809753:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   758.017735:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   758.225704:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   758.433698:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   758.641669:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   758.849646:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   759.057654:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   759.265627:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   759.473589:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   759.681579:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   759.889551:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   760.097535:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   760.305528:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   760.513500:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   760.721471:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   760.929463:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   761.137423:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   761.345427:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   761.553413:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   761.761392:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   761.969355:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   762.177327:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   762.385308:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   762.593298:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   762.801264:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:1-cgr  3623 [002]   763.009297:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   763.217226:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   763.425208:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   763.633214:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   763.841177:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   764.049161:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   764.257149:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   764.465121:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   764.673087:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   764.881069:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   765.089055:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   765.297054:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   765.505015:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   765.712995:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   765.920980:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   766.128945:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   766.336948:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   766.544915:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   766.752888:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   766.960866:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   767.168877:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   767.376856:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   767.584839:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   767.792792:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   768.000776:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   768.208753:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   768.416744:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   768.624724:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   768.832725:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   769.040668:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   769.248660:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   769.456629:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   769.665600:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   769.872605:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   770.080581:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   770.288557:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   770.496530:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   770.704518:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   770.912492:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   771.120491:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   771.328477:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   771.536445:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   771.744445:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   771.952413:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   772.160396:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   772.368350:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   772.576338:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   772.784326:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   772.992300:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   773.200284:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   773.408329:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   773.616230:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   773.824214:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   774.032190:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   774.240179:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   774.448149:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   774.656202:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   774.864115:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   775.072100:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   775.280098:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   775.488053:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   775.696041:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   775.904017:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   776.112004:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   776.319979:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   776.527962:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   776.735935:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   776.943965:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   777.151910:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   777.359874:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   777.567852:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   777.775837:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   777.983809:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   778.191801:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   778.399788:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   778.607755:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   778.815733:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   779.023719:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   779.231702:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   779.439677:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   779.647671:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   779.855690:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   780.063611:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   780.271590:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   780.479572:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   780.687562:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   780.895555:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   781.103552:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   781.311491:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   781.519466:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   781.727452:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   781.935436:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   782.143417:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   782.351406:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   782.559393:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   782.767350:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   782.975329:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   783.183311:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   783.391293:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   783.599282:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   783.807259:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   784.015228:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   784.223214:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   784.431196:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   784.639201:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   784.847174:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   785.055141:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   785.263143:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   785.471109:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   785.679155:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   785.887073:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   786.095033:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   786.303017:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   786.510999:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   786.719025:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   786.926958:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   787.134935:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   787.342949:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   787.550920:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   787.758876:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   787.966861:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   788.174833:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   788.382823:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   788.590798:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   788.798774:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   789.006777:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   789.214758:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   789.422741:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   789.630725:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   789.838701:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   790.046662:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   790.254670:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   790.462647:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   790.670599:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   790.878575:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   791.086576:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   791.294551:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   791.502515:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   791.710508:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   791.918493:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   792.126469:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   792.334457:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   792.542431:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   792.750396:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   792.958386:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   793.166361:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   793.374347:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   793.582311:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   793.790298:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   793.998292:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   794.206323:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   794.414254:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   794.622217:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   794.830199:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   795.038200:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   795.246161:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   795.454142:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   795.662122:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   795.870120:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   796.078072:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   796.286115:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   796.494084:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   796.702018:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   796.909998:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   797.117984:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   797.325961:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   797.533943:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   797.741942:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   797.949922:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   798.157885:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   798.365865:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   798.573842:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   798.781828:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   798.989853:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   799.197805:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   799.405753:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   799.613755:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   799.821766:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   800.029711:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   800.237697:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   800.445668:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   800.653650:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   800.861622:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   801.069601:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   801.277581:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   801.485583:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   801.693559:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   801.901539:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   802.109499:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   802.317513:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   802.525459:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   802.733437:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   802.941426:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   803.149420:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   803.357405:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   803.565381:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   803.773370:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   803.981333:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   804.189325:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   804.397309:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   804.605261:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   804.813261:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   805.021213:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   805.229203:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   805.437209:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   805.645160:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   805.853202:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   806.061117:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   806.269128:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   806.477102:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   806.685089:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   806.893069:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   807.101015:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   807.309008:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   807.516991:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   807.724982:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   807.932945:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   808.140949:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   808.348912:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   808.556902:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   808.764886:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   808.972866:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   809.180834:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   809.388827:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   809.596793:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   809.804787:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   810.012746:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   810.220730:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   810.428742:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   810.636693:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   810.844675:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   811.052665:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   811.260627:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   811.468603:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   811.676580:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   811.884578:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   812.092820:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   812.300510:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   812.508512:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   812.716486:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   812.924478:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.132477:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.340428:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.548411:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.756400:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.756404:      probe:mptcp_worker_L24: (ffffffffaeba46a0) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   813.756411: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0
 kworker/2:3+eve   242 [002]   813.964422:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   814.172346:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   814.380324:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   814.588315:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   814.796290:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   815.004274:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   815.212247:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   815.420230:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   815.628209:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   815.836191:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   816.044216:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   816.252152:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   816.460123:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   816.668110:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   816.876080:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   817.084061:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   817.292108:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   817.500022:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   817.708007:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   817.916009:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   818.123964:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   818.331951:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   818.539934:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   818.747924:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   818.955903:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   819.163872:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   819.371864:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   819.579822:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   819.787805:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   819.995800:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   820.203792:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   820.411746:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   820.619752:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   820.827708:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   821.035691:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   821.243685:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   821.451670:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   821.659634:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   821.867658:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   822.075645:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   822.283571:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   822.491553:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   822.699526:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   822.907518:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   823.115499:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   823.323478:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   823.531447:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   823.739450:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   823.947426:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   824.155382:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   824.363408:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   824.571361:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   824.779329:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   824.987319:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   825.195300:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   825.403308:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   825.611252:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   825.819242:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   826.027216:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   826.235234:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   826.443171:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   826.651151:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   826.859132:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   827.067116:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   827.275104:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   827.483068:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   827.691050:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   827.899028:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   828.107010:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   828.314999:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   828.522969:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   828.730954:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   828.938957:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   829.146919:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   829.354884:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   829.562894:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   829.770864:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   829.978840:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   830.187805:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   830.394792:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   830.602795:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   830.810745:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   831.018724:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   831.226733:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   831.434701:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   831.642687:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   831.850658:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   832.058625:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   832.266616:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   832.474596:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   832.682628:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   832.890559:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   833.098540:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   833.306506:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   833.514506:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   833.722495:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   833.930448:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   834.138435:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   834.346412:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   834.554413:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   834.762374:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   834.970371:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   835.178347:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   835.386312:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   835.594303:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   835.802280:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   836.010305:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   836.218241:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   836.426222:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   836.634208:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   836.842190:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   837.050149:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   837.258131:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   837.466112:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   837.674143:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   837.882084:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   838.090097:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   838.298046:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   838.506042:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   838.714009:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   838.921988:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   839.129960:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   839.337940:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   839.545915:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   839.753898:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   839.961877:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   840.169865:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   840.377866:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   840.585813:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   840.793790:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   841.001779:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   841.209767:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   841.417750:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   841.625722:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   841.833762:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   842.041697:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   842.249660:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   842.457645:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   842.665631:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   842.873606:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   843.081594:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   843.289565:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   843.497563:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   843.705522:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   843.913504:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   844.121493:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   844.329479:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   844.537449:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   844.745421:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   844.953615:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   845.161395:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   845.369372:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   845.577340:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   845.785342:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   845.993300:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   846.201282:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   846.409253:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   846.617239:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   846.825229:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   847.033201:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   847.241191:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   847.449267:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   847.657146:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   847.865137:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   848.073112:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   848.281088:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   848.489073:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   848.697046:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   848.905050:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   849.113032:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   849.321003:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   849.528953:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   849.736959:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   849.944927:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   850.152919:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   850.360890:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   850.568886:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   850.776888:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   850.984813:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   851.192814:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   851.400805:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   851.608778:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   851.816744:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   852.024770:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   852.232741:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   852.440748:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   852.648688:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   852.856655:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   853.064638:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   853.272603:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   853.480599:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   853.688553:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   853.896572:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   854.104534:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   854.312520:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   854.520467:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   854.728473:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   854.936477:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   855.144467:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   855.352417:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   855.560416:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   855.768393:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   855.976373:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   856.184345:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   856.392328:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   856.600301:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   856.808302:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   857.016271:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   857.224231:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   857.432228:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   857.640231:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   857.848180:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   858.056159:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   858.264130:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   858.472140:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   858.680117:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   858.888107:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   859.096074:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   859.304045:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   859.512050:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   859.720022:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   859.927987:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   860.135962:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   860.343929:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   860.551929:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   860.759909:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   860.967876:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   861.175856:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   861.383851:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   861.591809:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   861.799794:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   862.007806:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   862.215740:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   862.423731:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   862.631704:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   862.839680:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   863.047664:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   863.255674:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   863.463652:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   863.671597:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   863.884619:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   864.087569:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   864.295582:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   864.503506:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   864.711504:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   864.919470:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   865.127473:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   865.335439:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   865.543426:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   865.751454:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   865.959406:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   866.167358:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   866.375343:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   866.583345:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   866.791302:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   866.999314:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   867.207266:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   867.415238:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   867.623226:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   867.831203:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   868.039200:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   868.247160:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   868.455162:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   868.663137:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   868.871119:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   869.079091:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   869.287064:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   869.495057:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   869.703033:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   869.911011:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   870.118978:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   870.326987:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   870.534942:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   870.742937:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   870.950916:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   871.158895:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   871.366898:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   871.574839:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   871.787840:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   871.990821:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   872.198776:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   872.406786:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   872.615727:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   872.822741:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.030718:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.238686:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.446683:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.654684:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.862650:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.862654:      probe:mptcp_worker_L24: (ffffffffaeba46a0) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   873.862660: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0
 kworker/2:3+eve   242 [002]   874.070643:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   874.278589:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   874.486580:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   874.694557:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   874.902518:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   875.110729:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   875.318479:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   875.526459:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   875.734444:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   875.942429:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   876.150402:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   876.358400:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   876.566368:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   876.774345:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   876.982337:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   877.190316:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   877.398288:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   877.606256:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   877.814244:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   878.022234:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   878.230209:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   878.438193:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   878.646172:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   878.854194:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   879.062151:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   879.270105:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   879.478100:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   879.686076:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   879.894140:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   880.102029:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   880.310032:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   880.518005:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   880.725984:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   880.933952:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   881.141932:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   881.349909:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   881.557882:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   881.765895:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   881.973928:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   882.181818:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   882.389819:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   882.597777:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   882.805777:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   883.013763:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   883.221752:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   883.429711:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   883.637733:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   883.845664:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   884.053645:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   884.261635:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   884.469627:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   884.677595:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   884.885638:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   885.093567:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   885.301560:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   885.509531:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   885.717508:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   885.925490:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   886.133446:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   886.341446:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   886.549412:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   886.757398:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   886.965377:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   887.173369:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   887.381353:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   887.589361:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   887.797299:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   888.005267:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   888.213261:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   888.421216:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   888.629254:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   888.837219:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   889.045231:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   889.253168:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   889.461142:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   889.669103:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   889.877119:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   890.085096:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   890.293071:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   890.501042:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   890.709022:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   890.917020:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   891.130328:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   891.332951:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   891.540934:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   891.748925:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   891.956892:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   892.164898:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   892.372875:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   892.580841:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   892.788839:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   892.996799:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   893.204818:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   893.412765:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   893.620749:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   893.828752:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   894.036708:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   894.244672:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   894.452691:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   894.660645:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   894.868660:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   895.076658:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   895.284583:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   895.492558:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   895.700529:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   895.908539:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   896.116513:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   896.324498:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   896.532477:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   896.740566:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   896.948446:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   897.156435:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   897.364400:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   897.572374:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   897.780351:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   897.988338:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   898.196297:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   898.404283:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   898.612280:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   898.820239:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   899.028210:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   899.236192:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   899.444168:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   899.652156:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   899.860198:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   900.068125:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   900.276096:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   900.484067:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   900.692067:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   900.900037:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   901.108019:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   901.315986:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   901.523973:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   901.731949:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   901.939932:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   902.147933:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   902.355934:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   902.563921:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   902.771854:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   902.979848:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   903.187812:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   903.395828:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   903.603773:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   903.811755:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   904.019761:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   904.227738:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   904.435701:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   904.643703:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   904.851658:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   905.059671:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   905.267620:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   905.475593:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   905.683620:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   905.891565:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   906.099552:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   906.307521:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   906.515507:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   906.723470:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   906.931460:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   907.139428:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   907.347422:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   907.555398:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   907.763383:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   907.971358:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   908.179367:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   908.387309:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   908.595289:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   908.803306:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   909.011257:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   909.219261:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   909.427228:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   909.635215:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   909.843208:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   910.051167:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   910.259186:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   910.467144:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   910.675110:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   910.883111:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   911.091086:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   911.299078:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   911.507012:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   911.714985:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   911.922969:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   912.131943:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   912.338955:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   912.546930:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   912.754904:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   912.962904:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   913.170850:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   913.378830:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   913.586854:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   913.794790:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/2:3+eve   242 [002]   914.002769:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000

I hope that's useful.

Let me know if you also need the same from serverB (with the hung rsyncd process).

daire-byrne commented 1 year ago

Sorry, I should have attached that as a file - didn't realise how many lines there were...

pabeni commented 1 year ago

With such concise steps, how could I fail?!

IMHO it was fairly non trivial. Thanks a lot for the data collection!

So I grabbed the perf probes on serverA starting right before the "ifconfig ens256 up" on serverB. Then I proceeded to bring up the interface which hung the in progress mptcpize rsync. I waited a few minutes and then stopped the perf probe.


 kworker/2:3+eve   242 [002]   693.453136:      probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000
 kworker/1:0-mm_  2829 [001]   693.469860: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d807bd00 first=0xffff93b3f980ec00 ssk_state=0x8 subflows=0x1

1st subflow is closed.

kworker/1:0-mm_ 2829 [001] 693.469863: probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000 kworker/2:3+eve 242 [002] 693.473901: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x8 subflows=0x0

2nd subflow is closed

kworker/2:3+eve 242 [002] 693.473903: probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000 kworker/2:3+eve 242 [002] 693.507147: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0

3rd subflow (main one) is closed, the close timeout is started

[...]

kworker/2:3+eve 242 [002] 753.650160: probe:mptcp_worker_L23: (ffffffffaeba44b5) msk=0xffff93b3f62b8000

the close timeout expires

kworker/2:3+eve 242 [002] 753.650164: probe:mptcp_worker_L24: (ffffffffaeba46a0) msk=0xffff93b3f62b8000

the mptcp worker see the timeout and invokes mptcp_do_fastclose...

kworker/2:3+eve 242 [002] 753.650171: probe:__mptcp_close_ssk_L89: (ffffffffaeba3bbf) sk=0xffff93b3f62b8000 skc_state=0x1 next=0xffff93b3d82bda00 prev=0xffff93b3d82bda00 first=0xffff93b3f980ec00 ssk_state=0x7 subflows=0x0

... which walks again the subflows list. the main subflow is never deletes, so is still there even if already closed. Nothing to do, all good... ... except that the msk state is NOT moved into the TCP_CLOSE state (skc_state=0x1 == TCP_ESTABLISHED). Why? you are on 6.4.11, it lacks the upstream commit:

commit bbd49d114d5790a2dc3f67926cec3fc48b2d5d81 Author: Paolo Abeni pabeni@redhat.com Date: Tue Jun 20 18:30:21 2023 +0200

mptcp: consolidate transition to TCP_CLOSE in mptcp_do_fastclose()

which add the state transition to mptcp_do_fastclose. I forgot/did not notice the above was a pre-req for the tentative fix. Please additionally backport such commit, too, on top of the tentative fix. As I noted the posted revision contains an integer overflow making the fix not effective for the first 5'. I'll post soon a new revision, but even the already shared one should be good 5' after boot.

I hope that's useful.

Let me know if you also need the same from serverB (with the hung rsyncd process).

Indeed I think it was! thanks again for the debugging effort.

pabeni commented 1 year ago

... except that the msk state is NOT moved into the TCP_CLOSE state (skc_state=0x1 == TCP_ESTABLISHED). Why? you are on 6.4.11, it lacks the upstream commit:

commit bbd49d1 Author: Paolo Abeni pabeni@redhat.com Date: Tue Jun 20 18:30:21 2023 +0200

mptcp: consolidate transition to TCP_CLOSE in mptcp_do_fastclose()

Side notes:

daire-byrne commented 1 year ago

Right... that all makes sense now. I wanted to stick with v6.4.11 just to keep my testing consistent over the last couple of weeks.

I'm compiling a new version now and I'm fairly confident it will work this time. Sorry for the extra work, but we got there in the end!

So one thing I was still curious about was why bringing up the downed interface killed off the other remaining subflows too. Well, it looks like the act of bringing up the interface mangles the mptcp endpoints on serverB:

serverB # ip mptcp endpoint show
10.17.20.251 id 1 backup dev ens192 
10.17.21.251 id 2 signal dev ens224 
10.17.22.251 id 3 signal dev ens256 
serverB # ip link set ens256 down
serverB # ip mptcp endpoint show
10.17.20.251 id 1 backup dev ens192 
10.17.21.251 id 2 signal dev ens224 
10.17.22.251 id 3 signal dev ens256 
serverB # ip link set ens256 up
serverB # ip mptcp endpoint
10.17.21.251 id 3 subflow dev ens224 
10.17.22.251 id 4 subflow dev ens256 
10.17.20.251 id 5 subflow dev ens192

I guess I hadn't noticed that before and probably explains why the consistent flow on ens224 dies too.

I'm using EL8.7 (iproute-5.18).

pabeni commented 1 year ago

I guess I hadn't noticed that before and probably explains why the consistent flow on ens224 dies too.

I'm using EL8.7 (iproute-5.18).

How do you create the endpoints? do you use some 'network-scripts'? Do you have the mptcpd daemon installed/running? iproute by itself should not touch endpoints. Which NetworkManager version are you using? Starting from 1.4 it supports touching automatically mptcp endpoints.

daire-byrne commented 1 year ago

We do have the mptcpd package installed for mptcpize but it's not running the service. Our initial setup of the endpoints is just a script and systemd service.

It looks like EL8.7 does indeed come with NetworkManager-1.40 so I guess I need to figure out how to stop it trying to do anything with mptcp endpoints. Something like this by the looks of it:

by configuring the global connection defaults in a file /etc/NetworkManager/conf.d/95-no-mptcp.conf that contains

[connection.no-mptcp] # Configure a global connection default for the per-profile setting # "connection.mptcp-flags". 0x1 means "disabled". connection.mptcp-flags=0x1

pabeni commented 1 year ago

It looks like EL8.7 does indeed come with NetworkManager-1.40 so I guess I need to figure out how to stop it trying to do anything with mptcp endpoints. Something like this by the looks of it:

Looks good. Eventually you could consider integrating directly your setup into the NM configuration. This could help:

https://networkmanager.dev/docs/api/latest/nm-settings-nmcli.html#nm-settings-nmcli.property.connection.mptcp-flags

BTW this version of the fix should address the <not effective for 5' after reboot> problem: https://lore.kernel.org/mptcp/cover.1693565414.git.pabeni@redhat.com/T/#t

daire-byrne commented 1 year ago

And as hoped - we have success!

 kworker/3:29-xf  7475 [003]   866.000480:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/1:0-eve  6667 [001]   866.006271: probe:__mptcp_close_ssk_L89: (ffffffff9eba3a2f) sk=0xffff9024b04c8f80 skc_state=0x1 next=0xffff902484fb5200 prev=0xffff9024b755be00 first=0xffff9024a0b93f00 ssk_state=0x8 subflows=0x1
 kworker/1:0-eve  6667 [001]   866.006274:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   866.014273: probe:__mptcp_close_ssk_L89: (ffffffff9eba3a2f) sk=0xffff9024b04c8f80 skc_state=0x1 next=0xffff902484fb5200 prev=0xffff902484fb5200 first=0xffff9024a0b93f00 ssk_state=0x8 subflows=0x0
 kworker/3:29-xf  7475 [003]   866.014275:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   866.051516: probe:__mptcp_close_ssk_L89: (ffffffff9eba3a2f) sk=0xffff9024b04c8f80 skc_state=0x1 next=0xffff902484fb5200 prev=0xffff902484fb5200 first=0xffff9024a0b93f00 ssk_state=0x7 subflows=0x0
 kworker/3:29-xf  7475 [003]   866.051519:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   866.252065:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   866.460024:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
..
..
.. 
 kworker/3:29-xf  7475 [003]   926.158327:      probe:mptcp_worker_L23: (ffffffff9eba4325) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   926.158341:      probe:mptcp_worker_L24: (ffffffff9eba4510) msk=0xffff9024b04c8f80
 kworker/3:29-xf  7475 [003]   926.158348: probe:__mptcp_close_ssk_L89: (ffffffff9eba3a2f) sk=0xffff9024b04c8f80 skc_state=0x7 next=0xffff902484fb5200 prev=0xffff902484fb5200 first=0xffff9024a0b93f00 ssk_state=0x7 subflows=0x0
           rsync  7860 [003]   926.160334: probe:__mptcp_close_ssk_L89: (ffffffff9eba3a2f) sk=0xffff9024b04c8f80 skc_state=0x7 next=0xffff9024b04c9600 prev=0xffff9024b04c9600 first=0x0 ssk_state=0x0 subflows=0x0

Thanks for sticking with it and helping me through it :)

I will continue to monitor our production transfers with the kernel and see how it looks next week.

daire-byrne commented 1 year ago

I have not seen a single instance of hung/crashed/errored rsync transfers on our production servers for a few days now (and I would have expected to by now).

I think it's probably safe to say that this ticket and the other associated ones can now be resolved. Many thanks with all the help!

I'm still unsure while these all came to a head with the migration to v6.4 when most look like long standing issues that we must have been experiencing at a lower rate with v6.2. But I'm also fine with moving on and not worrying about it anymore :)

daire-byrne commented 1 year ago

I'm still unsure while these all came to a head with the migration to v6.4 when most look like long standing issues that we must have been experiencing at a lower rate with v6.2. But I'm also fine with moving on and not worrying about it anymore

It was still bugging me so I did a little more testing - when I first started using v6.4 I had also applied a patch from yet another cloudflare blog:

https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it

I reapplied that to my stable v6.4.11 kernel + your mptcp fixes and I quickly started seeing hung rsync transfers again. This time they seem to be hanging at the tcp subflow level. This patch is looking likely to end up in a future kernel release so I will have to retest again then with the final merged version.

So I think my flurry of bug reports were actually inspired from the high level of failure caused by this patch. I removed it quite early on and then we were left with the actual mptcp bugs that had been present at a much lower frequency for quite some time.

Either way, everything has been rock solid for the last week.

pabeni commented 1 year ago

I'm still unsure while these all came to a head with the migration to v6.4 when most look like long standing issues that we must have been experiencing at a lower rate with v6.2. But I'm also fine with moving on and not worrying about it anymore

It was still bugging me so I did a little more testing - when I first started using v6.4 I had also applied a patch from yet another cloudflare blog:

https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it

Thanks for the head-up! the relevant patch is already into 6.5, we will have to deal with it. AFAICS the new feature is protected by a sysctl knob, disabled by default.

I guess you tested after explicitly setting net.ipv4.tcp_shrink_window=1, am I correct?

On top of my head, I think such setting is not compatible at all with mptcp, IIRC the MPTCP RFC has some explicit discussion nothing that shrinking the subflow receive window will cause stalls.

Could you please double check that there are no problems with net.ipv4.tcp_shrink_window=0 ?

Thanks!