matttbe
commented
3 months ago It is maybe not linked, but MP_FAIL MP_RST is unstable: not on our side or NIPA, but on LKFT side:
# 105 MP_FAIL MP_RST
# Info: Test file (size 1024 KB) for client
# Info: Test file (size 1024 KB) for server
# nstat: history is aged out, resetting
# nstat: history is aged out, resetting
# write: Resource temporarily unavailable
# copyfd_io_poll: poll timed out (events: POLLIN 0, POLLOUT 4)
# nstat: history is aged out, resetting
# nstat: history is aged out, resetting
# [FAIL] client exit code 111, server 0
# Server ns stats
# TcpPassiveOpens 2 0.0
# TcpEstabResets 1 0.0
# TcpInSegs 236 0.0
# TcpOutSegs 387 0.0
# TcpOutRsts 1 0.0
# TcpExtDelayedACKs 5 0.0
# TcpExtTCPPureAcks 170 0.0
# TcpExtTCPLossProbes 2 0.0
# TcpExtTCPRcvCoalesce 42 0.0
# TcpExtTCPOrigDataSent 337 0.0
# TcpExtTCPDelivered 337 0.0
# MPTcpExtMPCapableSYNRX 1 0.0
# MPTcpExtMPCapableACKRX 1 0.0
# MPTcpExtMPTCPRetrans 155 0.0
# MPTcpExtMPJoinSynRx 1 0.0
# MPTcpExtMPJoinAckRx 1 0.0
# MPTcpExtDataCsumErr 1 0.0
# MPTcpExtOFOQueueTail 5 0.0
# MPTcpExtOFOQueue 6 0.0
# MPTcpExtOFOMerge 3 0.0
# MPTcpExtDuplicateData 8 0.0
# MPTcpExtMPFailTx 1 0.0
# MPTcpExtMPRstTx 1 0.0
# MPTcpExtSndWndShared 147 0.0
# MPTcpExtRcvWndShared 18 0.0
# Client ns stats
# TcpActiveOpens 2 0.0
# TcpEstabResets 1 0.0
# TcpInSegs 296 0.0
# TcpOutSegs 292 0.0
# TcpExtPruneCalled 9 0.0
# TcpExtDelayedACKs 138 0.0
# TcpExtTCPPureAcks 47 0.0
# TcpExtTCPLossProbes 1 0.0
# TcpExtTCPRcvCoalesce 2 0.0
# TcpExtTCPToZeroWindowAdv 145 0.0
# TcpExtTCPWantZeroWindowAdv 14 0.0
# TcpExtTCPOrigDataSent 120 0.0
# TcpExtTCPDelivered 121 0.0
# MPTcpExtMPCapableSYNTX 1 0.0
# MPTcpExtMPCapableSYNACKRX 1 0.0
# MPTcpExtMPJoinSynAckRx 1 0.0
# MPTcpExtDuplicateData 40 0.0
# MPTcpExtMPFailRx 1 0.0
# MPTcpExtMPRstRx 1 0.0
# MPTcpExtRcvPruned 187 0.0
# MPTcpExtRcvWndShared 161 0.0
#
# netns ns1-68845c7e-SwLcTs socket stat for 10104:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
# tcp ESTAB 0 64 [::ffff:10.0.1.1]:10104 [::ffff:10.0.1.2]:55762 timer:(on,290ms,0) ino:0 sk:1 cgroup:unreachable:6fb <->
# ts sack reno wscale:7,7 rto:293 rtt:92.865/41.324 ato:44 mss:1448 pmtu:1500 rcvmss:1420 advmss:1448 cwnd:82 bytes_sent:259838 bytes_acked:259774 bytes_received:155392 segs_out:383 segs_in:284 data_segs_out:338 data_segs_in:114 send 10228698bps lastsnd:3 lastrcv:32948 lastack:135 pacing_rate 20457392bps delivery_rate 149264bps delivered:338 busy:15422ms rwnd_limited:24ms(0.2%) sndbuf_limited:13851ms(89.8%) unacked:1 rcv_rtt:397.242 rcv_space:14600 rcv_ssthresh:64076 minrtt:5.722 snd_wnd:64 tcp-ulp-mptcp flags:Mec token:0000(id:0)/d89e6ca(id:0) seq:607429828340e720 sfseq:1f981 ssnoff:8ec6588a maplen:6580
# mptcp LAST-ACK 0 0 [::ffff:10.0.1.1]:10104 [::ffff:10.0.1.2]:55762 timer:(keepalive,59sec,0) ino:0 sk:2 cgroup:unreachable:6fb ---
# subflows:1 subflows_max:1 remote_key token:d89e6ca write_seq:60aea59871f2e1da snd_una:60aea59871eaac9b rcv_nxt:6074298283414ca1
# TcpPassiveOpens 2 0.0
# TcpEstabResets 1 0.0
# TcpInSegs 236 0.0
# TcpOutSegs 387 0.0
# TcpOutRsts 1 0.0
# TcpExtDelayedACKs 5 0.0
# TcpExtTCPPureAcks 170 0.0
# TcpExtTCPLossProbes 2 0.0
# TcpExtTCPRcvCoalesce 42 0.0
# TcpExtTCPOrigDataSent 337 0.0
# TcpExtTCPDelivered 337 0.0
# MPTcpExtMPCapableSYNRX 1 0.0
# MPTcpExtMPCapableACKRX 1 0.0
# MPTcpExtMPTCPRetrans 155 0.0
# MPTcpExtMPJoinSynRx 1 0.0
# MPTcpExtMPJoinAckRx 1 0.0
# MPTcpExtDataCsumErr 1 0.0
# MPTcpExtOFOQueueTail 5 0.0
# MPTcpExtOFOQueue 6 0.0
# MPTcpExtOFOMerge 3 0.0
# MPTcpExtDuplicateData 8 0.0
# MPTcpExtMPFailTx 1 0.0
# MPTcpExtMPRstTx 1 0.0
# MPTcpExtSndWndShared 147 0.0
# MPTcpExtRcvWndShared 18 0.0
#
# netns ns2-68845c7e-SwLcTs socket stat for 10104:
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Failed to find cgroup2 mount
# Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
# tcp ESTAB 17920 0 10.0.1.2:55762 10.0.1.1:10104 ino:0 sk:1001 cgroup:unreachable:6fb <->
# ts sack reno wscale:7,7 rto:1031 rtt:443.443/55.781 ato:128 mss:1448 pmtu:1500 rcvmss:1420 advmss:1448 cwnd:104 bytes_sent:155392 bytes_acked:155393 bytes_received:259838 segs_out:284 segs_in:384 data_segs_out:114 data_segs_in:338 send 2716778bps lastsnd:33216 lastrcv:45 lastack:45 pacing_rate 5433552bps delivery_rate 626880bps delivered:115 busy:1346ms rwnd_limited:166ms(12.3%) rcv_rtt:156.689 rcv_space:14480 rcv_ssthresh:64088 minrtt:7.099 snd_wnd:64128 tcp-ulp-mptcp flags:Mmecv token:0000(id:0)/23631e7a(id:0) seq:60aea59871eaa183 sfseq:3a5e7 ssnoff:62d4f68d maplen:b58
# mptcp FIN-WAIT-2 158046 0 10.0.1.2:55762 10.0.1.1:10104 timer:(keepalive,25sec,0) ino:0 sk:1002 cgroup:unreachable:6fb ---
# subflows_max:1 remote_key token:23631e7a write_seq:6074298283414ca1 snd_una:6074298283414ca1 rcv_nxt:60aea59871eaac9b
# TcpActiveOpens 2 0.0
# TcpEstabResets 1 0.0
# TcpInSegs 296 0.0
# TcpOutSegs 292 0.0
# TcpExtPruneCalled 9 0.0
# TcpExtDelayedACKs 138 0.0
# TcpExtTCPPureAcks 47 0.0
# TcpExtTCPLossProbes 1 0.0
# TcpExtTCPRcvCoalesce 2 0.0
# TcpExtTCPToZeroWindowAdv 145 0.0
# TcpExtTCPWantZeroWindowAdv 14 0.0
# TcpExtTCPOrigDataSent 120 0.0
# TcpExtTCPDelivered 121 0.0
# MPTcpExtMPCapableSYNTX 1 0.0
# MPTcpExtMPCapableSYNACKRX 1 0.0
# MPTcpExtMPJoinSynAckRx 1 0.0
# MPTcpExtDuplicateData 40 0.0
# MPTcpExtMPFailRx 1 0.0
# MPTcpExtMPRstRx 1 0.0
# MPTcpExtRcvPruned 187 0.0
# MPTcpExtRcvWndShared 161 0.0
# Info: 1 corrupted pkts
# syn [ OK ]
# synack [ OK ]
# ack [ OK ]
# sum [ OK ]
# csum [ OK ]
# ftx [ OK ]
# failrx [ OK ]
# rtx [ OK ]
# rstrx [ OK ]
# itx [ OK ]
# infirx [ OK ]Log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240517/testrun/23977459/suite/kselftest-net-mptcp/test/net_mptcp_mptcp_join_sh_-_mptcp_join_MP_FAIL_MP_RST/log History: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240517/testrun/23977459/suite/kselftest-net-mptcp/test/net_mptcp_mptcp_join_sh_-_mptcp_join_MP_FAIL_MP_RST/history/
It looks like we didn't have this issue recently on our side, but the NetDev CI got it a few times, e.g.
https://netdev-3.bots.linux.dev/vmksft-mptcp-dbg/results/594621/1-mptcp-join-sh/stdout
Or:
Or:
Even with a non debug kernel config (rarer):
https://netdev-3.bots.linux.dev/vmksft-mptcp/results/582921/1-mptcp-join-sh/stdout