multipath-tcp / mptcp_net-next

Development version of the Upstream MultiPath TCP Linux kernel 🐧
https://mptcp.dev
Other
255 stars 36 forks source link

selftests: `mptcp_connect.sh`: disconnect test timed out #499

Open matttbe opened 3 weeks ago

matttbe commented 3 weeks ago

This issue has been seen on NIPA, with a debug kernel config: the disconnect test in v4 with MPTCP on both side failed with a timeout:

# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001      ) MPTCP     copyfd_io_poll: poll timed out (events: POLLIN 1, POLLOUT 4)
# copyfd_io_poll: poll timed out (events: POLLIN 1, POLLOUT 4)
# main_loop_s: timed out
# (duration 60607ms) [FAIL] client exit code 2, server 2
# 
# netns ns1-H3axWx socket stat for 20001:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State      Recv-Q Send-Q Local Address:Port  Peer Address:Port                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
# tcp   ESTAB      0      0           10.0.1.1:20001     10.0.1.1:49604 ino:0 sk:1006 cgroup:unreachable:1 <->
#  ts sack cubic wscale:7,7 rto:201 rtt:0.302/0.1 ato:40 mss:32768 pmtu:65535 rcvmss:32740 advmss:65483 cwnd:10 bytes_sent:1223524 bytes_acked:1223524 bytes_received:1217280 segs_out:89 segs_in:91 data_segs_out:56 data_segs_in:57 send 8680264901bps lastsnd:60772 lastrcv:60771 lastack:60771 pacing_rate 17353347120bps delivery_rate 2978909088bps delivered:57 busy:13ms rcv_rtt:0.779 rcv_space:65483 rcv_ssthresh:65536 minrtt:0.083 tcp-ulp-mptcp flags:Mec token:0000(id:0)/719a9338(id:0) seq:a2cdd16f5e1643f8 sfseq:1292c9 ssnoff:b83253cc maplen:38
# mptcp FIN-WAIT-1 0      0           10.0.1.1:20001     10.0.1.1:49604 timer:(keepalive,59sec,0) ino:0 sk:1007 cgroup:unreachable:1 ---
#  subflows_max:2 remote_key token:719a9338 write_seq:11861448483224381307 snd_una:11861448483222639326 rcv_nxt:11731262880620102704 bytes_sent:1223524 bytes_received:1217280 bytes_acked:1223524 subflows_total:1 last_data_sent:60783 last_data_recv:60782 last_ack_recv:60782                                                                                                                                                                                                                                                       
# TcpActiveOpens                  1                  0.0
# TcpPassiveOpens                 1                  0.0
# TcpInSegs                       181                0.0
# TcpOutSegs                      180                0.0
# TcpRetransSegs                  1                  0.0
# TcpExtDelayedACKLost            1                  0.0
# TcpExtTCPPureAcks               66                 0.0
# TcpExtTCPLossProbes             1                  0.0
# TcpExtTCPDSACKOldSent           1                  0.0
# TcpExtTCPDSACKRecv              1                  0.0
# TcpExtTCPFromZeroWindowAdv      30                 0.0
# TcpExtTCPToZeroWindowAdv        32                 0.0
# TcpExtTCPWantZeroWindowAdv      9                  0.0
# TcpExtTCPOrigDataSent           112                0.0
# TcpExtTCPDelivered              114                0.0
# TcpExtTCPDSACKRecvSegs          1                  0.0
# MPTcpExtMPCapableSYNRX          1                  0.0
# MPTcpExtMPCapableSYNTX          1                  0.0
# MPTcpExtMPCapableSYNACKRX       1                  0.0
# MPTcpExtMPCapableACKRX          1                  0.0
# 
# netns ns1-H3axWx socket stat for 20001:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State      Recv-Q Send-Q Local Address:Port  Peer Address:Port                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
# tcp   ESTAB      0      0           10.0.1.1:49604     10.0.1.1:20001 ino:0 sk:3005 cgroup:unreachable:1 <->
#  ts sack cubic wscale:7,7 rto:201 rtt:0.723/0.563 ato:40 mss:32768 pmtu:65535 rcvmss:32740 advmss:65483 cwnd:10 bytes_sent:1217336 bytes_retrans:56 bytes_acked:1217281 bytes_received:1223524 segs_out:91 segs_in:90 data_segs_out:57 data_segs_in:56 send 3625781466bps lastsnd:61194 lastrcv:61194 lastack:61193 pacing_rate 7247803696bps delivery_rate 918728968bps delivered:58 busy:24ms retrans:0/1 dsack_dups:1 rcv_rtt:0.53 rcv_space:65495 rcv_ssthresh:65536 minrtt:0.132 tcp-ulp-mptcp flags:Mmec token:0000(id:0)/56769fbe(id:0) seq:a49c548d0414d2a6 sfseq:12ab2d ssnoff:c291bfb0 maplen:38
# mptcp FIN-WAIT-1 0      0           10.0.1.1:49604     10.0.1.1:20001 timer:(keepalive,28sec,0) ino:0 sk:3006 cgroup:unreachable:1 ---
#  subflows_max:2 remote_key token:56769fbe write_seq:11731262880621834545 snd_una:11731262880620102704 rcv_nxt:11861448483222639326 bytes_sent:1217280 bytes_received:1223524 bytes_acked:1217280 subflows_total:1 last_data_sent:61195 last_data_recv:61195 last_ack_recv:61194                                                                                                                                                                                                                                                                                                 
# 
# 64 ns1 MPTCP -> ns1 (dead:beef:1::1:20002) MPTCP     (duration   763ms) [ OK ]
# 65 ns1 MPTCP -> ns1 (dead:beef:1::1:20003) TCP       (duration   584ms) [ OK ]
# 66 ns1 TCP   -> ns1 (dead:beef:1::1:20004) MPTCP     (duration   569ms) [ OK ]
# [FAIL] Tests of the full disconnection have failed

From: contest - stdout

Note that the test has been re-run, and the test passed the second time.

matttbe commented 1 week ago

It looks like we just got it a second time (after 64 other attempts without issues):

# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001      ) MPTCP     copyfd_io_poll: poll timed out (events: POLLIN 1, POLLOUT 4)
# copyfd_io_poll: poll timed out (events: POLLIN 1, POLLOUT 4)
# main_loop_s: timed out
# (duration 60609ms) [FAIL] client exit code 2, server 2
# 
# netns ns1-P3Kvbz socket stat for 20001:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State      Recv-Q Send-Q Local Address:Port  Peer Address:Port                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
# tcp   ESTAB      0      0           10.0.1.1:20001     10.0.1.1:43290 ino:0 sk:1007 cgroup:unreachable:1 <->
#  ts sack cubic wscale:7,7 rto:201 rtt:0.439/0.265 ato:40 mss:32768 pmtu:65535 rcvmss:32740 advmss:65483 cwnd:10 bytes_sent:1688748 bytes_acked:1688748 bytes_received:1570404 segs_out:131 segs_in:123 data_segs_out:94 data_segs_in:88 send 5971389522bps lastsnd:60804 lastrcv:60804 lastack:60803 pacing_rate 11929192256bps delivery_rate 714288824bps delivered:95 busy:20ms rcv_rtt:0.8 rcv_space:65483 rcv_ssthresh:65536 minrtt:0.085 tcp-ulp-mptcp flags:Mec token:0000(id:0)/e7baab78(id:0) seq:d8de468b32ba23f7 sfseq:17f62d ssnoff:d63f6e86 maplen:38
# mptcp FIN-WAIT-1 0      0           10.0.1.1:20001     10.0.1.1:43290 timer:(keepalive,59sec,0) ino:0 sk:3004 cgroup:unreachable:1 ---
#  subflows_max:2 remote_key token:e7baab78 write_seq:12549881151337605123 snd_una:12549881151335869614 rcv_nxt:15627005320687658031 bytes_sent:1688748 bytes_received:1570404 bytes_acked:1688748 subflows_total:1 last_data_sent:60815 last_data_recv:60815 last_ack_recv:60814                                                                                                                                                                                                                                                        
# TcpActiveOpens                  1                  0.0
# TcpPassiveOpens                 1                  0.0
# TcpInSegs                       255                0.0
# TcpOutSegs                      255                0.0
# TcpExtTCPPureAcks               71                 0.0
# TcpExtTCPFromZeroWindowAdv      36                 0.0
# TcpExtTCPToZeroWindowAdv        38                 0.0
# TcpExtTCPWantZeroWindowAdv      19                 0.0
# TcpExtTCPOrigDataSent           182                0.0
# TcpExtTCPDelivered              183                0.0
# MPTcpExtMPCapableSYNRX          1                  0.0
# MPTcpExtMPCapableSYNTX          1                  0.0
# MPTcpExtMPCapableSYNACKRX       1                  0.0
# MPTcpExtMPCapableACKRX          1                  0.0
# 
# netns ns1-P3Kvbz socket stat for 20001:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State      Recv-Q Send-Q Local Address:Port  Peer Address:Port                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
# tcp   ESTAB      0      0           10.0.1.1:43290     10.0.1.1:20001 ino:0 sk:3005 cgroup:unreachable:1 <->
#  ts sack cubic wscale:7,7 rto:201 rtt:0.381/0.214 ato:40 mss:32768 pmtu:65535 rcvmss:32740 advmss:65483 cwnd:10 bytes_sent:1570404 bytes_acked:1570405 bytes_received:1688748 segs_out:123 segs_in:132 data_segs_out:88 data_segs_in:94 send 6880419948bps lastsnd:61218 lastrcv:61218 lastack:61218 pacing_rate 13751816392bps delivery_rate 2148721304bps delivered:89 busy:22ms rcv_rtt:0.793 rcv_space:65495 rcv_ssthresh:65536 minrtt:0.075 tcp-ulp-mptcp flags:Mmec token:0000(id:0)/698c1c2a(id:0) seq:ae2a226a3ebeac76 sfseq:19c475 ssnoff:38f53772 maplen:38
# mptcp FIN-WAIT-1 0      0           10.0.1.1:43290     10.0.1.1:20001 timer:(keepalive,28sec,0) ino:0 sk:3006 cgroup:unreachable:1 ---
#  subflows_max:2 remote_key token:698c1c2a write_seq:15627005320689413580 snd_una:15627005320687658031 rcv_nxt:12549881151335869614 bytes_sent:1570404 bytes_received:1688748 bytes_acked:1570404 subflows_total:1 last_data_sent:61221 last_data_recv:61220 last_ack_recv:61220                                                                                                                                                                                                                                                            
# 
# 64 ns1 MPTCP -> ns1 (dead:beef:1::1:20002) MPTCP     (duration   942ms) [ OK ]
# 65 ns1 MPTCP -> ns1 (dead:beef:1::1:20003) TCP       (duration   750ms) [ OK ]
# 66 ns1 TCP   -> ns1 (dead:beef:1::1:20004) MPTCP     (duration   718ms) [ OK ]
# [FAIL] Tests of the full disconnection have failed

stdout