search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
889 stars 336 forks source link

Client-Server MPTCP initialization failed #479

Closed ArthurCChen closed 2 years ago

ArthurCChen commented 2 years ago

Hi, I got a Ubuntu 18.04LTS PC as the client and an Amazon EC2 with Ubuntu 18.04LTS as the Server. I deployed MPTCP Stable release v0.95.2 on both successfully(both pass the curl http://www.multipath-tcp.org, the client sometimes may fail, see below, this part is strange, but I don't know why.).

(base) client@lab:~$ dmesg | grep MPTCP
[    0.376045] MPTCP: Stable release v0.95.2
(base) client@lab:~$ journalctl -k | grep -i mptcp
Apr 20 21:18:36 lab kernel: MPTCP: Stable release v0.95.2
(base) client@lab:~/src/tracebox$ curl http://www.multipath-tcp.org
Nay, Nay, Nay, your have an old computer that does not speak MPTCP. Shame on you!
(base) client@lab:~/src/tracebox$ curl http://www.multipath-tcp.org
Yay, you are MPTCP-capable! You can now rest in peace.
(base) client@lab:~/src/tracebox$ curl http://www.multipath-tcp.org
Yay, you are MPTCP-capable! You can now rest in peace.
(base) client@lab:~/src/tracebox$ curl http://www.multipath-tcp.org
Yay, you are MPTCP-capable! You can now rest in peace.
(base) client@lab:~/src/tracebox$ curl http://www.multipath-tcp.org
Nay, Nay, Nay, your have an old computer that does not speak MPTCP. Shame on you!

The client is with two interfaces, and the routing is automatically configured.

I'm hoping to see that any TCP communication between them turns to MPTCP, however, according to my Wireshark, only the first few packets of client requests are MPTCP, others are all TCP.

So I think it must be a failed negotiation and initialization, as two subflows are not established.

Firstly, I think it's because of the middlebox. So I tested with Tracebox:

(base) client@lab:~/src/tracebox$ sudo tracebox -p 'IP/tcp{dst=443}/MPCAPABLE' 184.169.218.6
tracebox to 184.169.218.6 (184.169.218.6): 64 hops max
1: _gateway (10.33.0.1) 1ms TCP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable 
2: 169.236.78.4 (169.236.78.4) 12ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
3: 10.7.1.61 (10.7.1.61) 1ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
4: 10.7.2.2 (10.7.2.2) 1ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable 
5: 10.7.2.18 (10.7.2.18) 26ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
6: dc-tri-agg2--uc-merced.cenic.net (137.164.23.137) 29ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable 
7: dc-tri-agg2--frg-agg4-100g.cenic.net (137.164.11.100) 6ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable [Extra headers: ICMPExtension ICMPExtensionObject ICMPExtensionMPLS ] 
8: 99.83.69.196 (99.83.69.196) 86ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
9: 52.93.141.159 (52.93.141.159) 82ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable [Extra headers: ICMPExtension ICMPExtensionObject ICMPExtensionMPLS ] 
10: 150.222.111.127 (150.222.111.127) 61ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
11: *
12: 150.222.30.46 (150.222.30.46) 64ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable [Extra headers: ICMPExtension ICMPExtensionObject ICMPExtensionMPLS ] 
13: 150.222.30.175 (150.222.30.175) 46ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
14: 150.222.31.100 (150.222.31.100) 11ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable [Extra headers: ICMPExtension ICMPExtensionObject ICMPExtensionMPLS ICMPExtensionMPLS ] 
15: 150.222.31.113 (150.222.31.113) 74ms TCP::CheckSum IP::TTL IP::CheckSum +TCPOptionPad -TCPOptionMPTCPCapable [Extra headers: ICMPExtension ICMPExtensionObject ICMPExtensionMPLS ] 
16: 15.230.36.130 (15.230.36.130) 9053ms  [PARTIAL] +PartialTCP IP::TTL IP::CheckSum 
17: *
18: *
19: *
20: *
21: *
22: *
23: ec2-184-169-218-6.us-west-1.compute.amazonaws.com (184.169.218.6) 9045ms TCP::SrcPort TCP::DstPort TCP::SeqNumber TCP::AckNumber TCP::DataOffset TCP::Flags TCP::WindowsSize TCP::CheckSum IP::TotalLength IP::Identification IP::TTL IP::CheckSum IP::SourceIP IP::DestinationIP -TCPOptionMPTCPCapable

Here 184.169.218.6 is my server.

Some other logs from Server side:

ubuntu@ip-172-31-16-143:~$ sysctl net.mptcp
net.mptcp.mptcp_binder_gateways = 
net.mptcp.mptcp_checksum = 1
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = roundrobin
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0
ubuntu@ip-172-31-16-143:~$ cat /proc/net/mptcp_fullmesh
Index, Address-ID, Backup, IP-address, if-idx
IPv4, next v4-index: 3
2, 3, 0, 172.31.16.143, 2
IPv6, next v6-index: 0

and Client side:

(base) client@lab:~$ sysctl net.mptcp
net.mptcp.mptcp_binder_gateways = 
net.mptcp.mptcp_checksum = 1
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = roundrobin
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0
(base) client@lab:~$ cat /proc/net/mptcp_fullmesh
Index, Address-ID, Backup, IP-address, if-idx
IPv4, next v4-index: 5
2, 3, 0, 192.168.123.1, 5
3, 4, 0, 10.33.23.188, 4
4, 5, 0, 192.168.1.3, 3
IPv6, next v6-index: 2
0, 8, 0, 2600:1010:b16c:ebb3:38cb:9776:9386:2caf, 3
1, 9, 0, 2600:1010:b16c:ebb3:6c1a:3e63:cec0:1b0d, 3

So what may be the problem and what should I do to modify it?

ArthurCChen commented 2 years ago

tcpdump | grep mptcp on the server and on one interface of the client shows that there are MPTCP available: (To generate MPTCP packets to be monitored, I kept doing curl http://www.multipath-tcp.org in the background.)

(base) client@lab:~$ sudo tcpdump -n -i wlp4s0 tcp | grep mptcp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:37:18.301949 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.49868 > 2607:f8b0:4005:812::2003.443: Flags [S], seq 1964793783, win 64296, options [mss 1368,sackOK,TS val 621119730 ecr 0,nop,wscale 7,mptcp capable csum {0x694789a3c914e2c0}], length 0
21:37:18.552207 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.49870 > 2607:f8b0:4005:812::2003.443: Flags [S], seq 3085643651, win 64296, options [mss 1368,sackOK,TS val 621119981 ecr 0,nop,wscale 7,mptcp capable csum {0xfa5a0fe787178352}], length 0
21:37:55.545039 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58300 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 3428591253, win 64296, options [mss 1368,sackOK,TS val 2068553499 ecr 0,nop,wscale 7,mptcp capable csum {0x6442290b1138d868}], length 0
21:37:56.393472 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58300: Flags [S.], seq 4255087202, ack 3428591254, win 28160, options [mss 1440,sackOK,TS val 4121092459 ecr 2068553499,nop,wscale 7,mptcp capable csum {0x59c694d4dd8a3803}], length 0
21:37:57.560367 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58304 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 2973371154, win 64296, options [mss 1368,sackOK,TS val 2068555515 ecr 0,nop,wscale 7,mptcp capable csum {0x585bad044579ec82}], length 0
21:37:57.962445 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58304: Flags [S.], seq 879574233, ack 2973371155, win 28160, options [mss 1440,sackOK,TS val 4121093891 ecr 2068555515,nop,wscale 7,mptcp capable csum {0x208940714fe65362}], length 0
21:38:01.921021 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58308 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 2044975270, win 64296, options [mss 1368,sackOK,TS val 2068559878 ecr 0,nop,wscale 7,mptcp capable csum {0xad57c6ebd861d417}], length 0
21:38:02.485883 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58308: Flags [S.], seq 560489339, ack 2044975271, win 28160, options [mss 1440,sackOK,TS val 4121098546 ecr 2068559878,nop,wscale 7,mptcp capable csum {0x8d7080f60e0415b3}], length 0
21:38:04.684879 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58312 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 3670445154, win 64296, options [mss 1368,sackOK,TS val 2068562643 ecr 0,nop,wscale 7,mptcp capable csum {0xd5f957f4f2264f8b}], length 0
21:38:05.130378 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58312: Flags [S.], seq 3873911719, ack 3670445155, win 28160, options [mss 1388,sackOK,TS val 4121101031 ecr 2068562643,nop,wscale 7,mptcp capable csum {0x1308b1ca6673cab6}], length 0
21:38:06.020358 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 290406786, win 64296, options [mss 1368,sackOK,TS val 2068563979 ecr 0,nop,wscale 7,mptcp capable csum {0x365470580a79b239}], length 0
21:38:06.358415 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316: Flags [S.], seq 1690924945, ack 290406787, win 28160, options [mss 1388,sackOK,TS val 4121102356 ecr 2068563979,nop,wscale 7,mptcp capable csum {0x68e47fd0c51731a5}], length 0
21:38:06.358501 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp capable csum {0x365470580a79b239,0x68e47fd0c51731a5},mptcp dss ack 3722348309], length 0
21:38:06.358529 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp add-addr id 2 169.236.180.156,mptcp dss ack 3722348309], length 0
21:38:06.358537 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp add-addr id 3 192.168.123.1,mptcp dss ack 3722348309], length 0
21:38:06.358544 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp add-addr id 4 10.33.23.188,mptcp dss ack 3722348309], length 0
21:38:06.358551 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp add-addr id 5 192.168.1.3,mptcp dss ack 3722348309], length 0
21:38:06.358559 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 1, win 503, options [nop,nop,TS val 2068564317 ecr 4121102356,mptcp add-addr id 8 2600:1010:b16c:ebb3:38cb:9776:9386:2caf,mptcp dss ack 3722348309], length 0
21:38:06.358721 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [P.], seq 1:86, ack 1, win 503, options [nop,nop,TS val 2068564318 ecr 4121102356,mptcp dss ack 3722348309 seq 27834557 subseq 1 len 85 csum 0xb75e], length 85: HTTP: GET / HTTP/1.1
21:38:06.602860 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316: Flags [.], ack 1, win 220, options [nop,nop,TS val 4121102674 ecr 2068564318,mptcp add-addr id 2 130.104.228.140,mptcp dss ack 27834557], length 0
21:38:06.602898 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316: Flags [.], ack 86, win 220, options [nop,nop,TS val 4121102674 ecr 2068564318,mptcp dss ack 27834642], length 0
21:38:06.603105 IP 192.168.1.3.51525 > 130.104.228.140.80: Flags [S], seq 3268237647, win 65236, options [mss 1388,sackOK,TS val 1179773116 ecr 0,nop,wscale 7,mptcp join id 5 token 0xf35bbfce nonce 0x2c797cde], length 0
21:38:06.603155 IP6 2600:1010:b16c:ebb3:38cb:9776:9386:2caf.53811 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [S], seq 727398278, win 64296, options [mss 1368,sackOK,TS val 4225380677 ecr 0,nop,wscale 7,mptcp join id 8 token 0xf35bbfce nonce 0x8ad0b440], length 0
21:38:06.609811 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316: Flags [P.], seq 1:260, ack 86, win 220, options [nop,nop,TS val 4121102680 ecr 2068564318,mptcp dss ack 27834642 seq 3722348309 subseq 1 len 259 csum 0x12f9], length 259: HTTP: HTTP/1.1 200 OK
21:38:06.609869 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 260, win 501, options [nop,nop,TS val 2068564569 ecr 4121102680,mptcp dss ack 3722348568], length 0
21:38:06.610098 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [F.], seq 86, ack 260, win 502, options [nop,nop,TS val 2068564569 ecr 4121102680,mptcp dss fin ack 3722348568 seq 27834642 subseq 0 len 1 csum 0xa1f1], length 0
21:38:06.975165 IP 130.104.228.140.80 > 192.168.1.3.51525: Flags [S.], seq 3060240310, ack 3268237648, win 28560, options [mss 1388,sackOK,TS val 28760478 ecr 1179773116,nop,wscale 7,mptcp join id 2 hmac 0xd91df594322ab380 nonce 0xb451cba9], length 0
21:38:06.975246 IP 192.168.1.3.51525 > 130.104.228.140.80: Flags [.], ack 1, win 1003, options [nop,nop,TS val 1179773488 ecr 28760478,mptcp join hmac 0x2572b05b9d8eac53d2da3dd73707e886410d574e], length 0
21:38:06.975276 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316: Flags [F.], seq 260, ack 87, win 220, options [nop,nop,TS val 4121102893 ecr 2068564569,mptcp dss fin ack 27834643 seq 3722348568 subseq 0 len 1 csum 0x77b0], length 0
21:38:06.975338 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 261, win 1003, options [nop,nop,TS val 2068564934 ecr 4121102893,mptcp dss ack 3722348569], length 0
21:38:06.975354 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.58316 > 2001:6a8:308f:8:0:82ff:fe68:e48c.80: Flags [.], ack 261, win 1003, options [nop,nop,TS val 2068564934 ecr 4121102893,mptcp dss ack 3722348569], length 0
21:38:07.075633 IP6 2001:6a8:308f:8:0:82ff:fe68:e48c.80 > 2600:1010:b16c:ebb3:38cb:9776:9386:2caf.53811: Flags [S.], seq 90542463, ack 727398279, win 28160, options [mss 1388,sackOK,TS val 945190066 ecr 4225380677,nop,wscale 7,mptcp join id 8 hmac 0x1b6e268a64565b19 nonce 0x96760062], length 0
21:38:12.513785 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.46336 > 2607:f8b0:4005:80b::2004.443: Flags [S], seq 1617955419, win 64296, options [mss 1368,sackOK,TS val 1784335943 ecr 0,nop,wscale 7,mptcp capable csum {0x5c574464c1a4a888}], length 0
21:38:12.514073 IP6 2600:1010:b16c:ebb3:6d5f:b9b3:25ce:437d.46338 > 2607:f8b0:4005:80b::2004.443: Flags [S], seq 2704162416, win 64296, options [mss 1368,sackOK,TS val 1784335944 ecr 0,nop,wscale 7,mptcp capable csum {0x90a5680993056529}], length 0
^C99 packets captured
99 packets received by filter
0 packets dropped by kernel
ArthurCChen commented 2 years ago

Nothing Modified but it passed. So I close this issue. Thanks all the same!

The reason is that after I moved the experiment site, middleboxes seem to disappear.