search

multipath-tcp / mptcp_net-next

Development version of the Upstream MultiPath TCP Linux kernel 🐧
https://mptcp.dev
Other
278 stars 41 forks source link

Subflows won't be used when the interface for the main flow is a subinterface #505

Open arinc9 opened 3 months ago

arinc9 commented 3 months ago

The default route is via 192.168.13.1 dev enp3s0.3.

I've set up the subflow I want to use:

$ sudo ip mp e
192.168.14.2 id 2 subflow dev enp3s0.4

Then described the gateway for the subflow path:

sudo ip rule add from 192.168.14.2 table 100
sudo ip route add default via 192.168.14.1 table 100

The subflow won't be used:

$ mptcpize run iperf3 -c hostIPv4
[       CREATED] token=20b1293a remid=0 locid=0 saddr4=192.168.13.2 daddr4=hostIPv4 sport=58362 dport=5201
Connecting to host hostIPv4, port 5201
[       CREATED] token=973e67fc remid=0 locid=0 saddr4=192.168.13.2 daddr4=hostIPv4 sport=58366 dport=5201
[  5] local 192.168.13.2 port 58366 connected to hostIPv4 port 5201

enp3s0.3 and enp3s0.4 are VLAN interfaces so they are enp3s0.3@enp3s0 and enp3s0.4@enp3s0. I've noticed that if the default route is a subinterface, subflows won't be created. I've also experienced this issue with DSA interfaces hence my belief that specifically subinterfaces, not just VLAN interfaces, are affected.

asirir commented 3 months ago

Hi,

I wonder if you need to set src on your routing table entries? Like:

hostIPv4 via 192.168.13.1 dev enp3s0.3 src <source_ip>

My setup is different (not fullmesh) but I do set the source IP to make the routing more specific.

arinc9 commented 3 months ago

I don't think src is necessary as there is only one IPv4 address assigned to enp3s0.3. I've tested it anyway; it doesn't change the current behaviour.

sudo ip route del hostIPv4 via 192.168.13.1 dev enp3s0.3
sudo ip route add hostIPv4 via 192.168.13.1 dev enp3s0.3 src 192.168.13.2
asirir commented 3 months ago

What's the mptcp configuration on hostIPv4? (I'm assuming the setup you describe in the question is just for the host from which you are running the iperf3 command)

arinc9 commented 3 months ago

The host acting as an iperf3 server has net.mptcp.enabled=1 and mptcpize run iperf3 -s is running. No mptcp endpoint is configured. The behaviour is the same if the default route is described as a signal path:

$ sudo ip mp e                                  
hostIPv4 id 1 signal dev eth0

Both hosts run Ubuntu's 6.8.0-35-generic kernel.

asirir commented 3 months ago

Hi,

How about the MPTCP limits on each host? that is, ip mptcp limits show - I don't think you need a signal endpoint on the remote host in this case, but I think you need at least subflow 1 add_addr_accepted 0 setup on both ends to make this work.

Also, do you need fullmesh endpoints on your source server? If you only need an additional subflow betwen the two hosts, a subflow endpoint would do no?

arinc9 commented 3 months ago

The limits are fine. I can see that subflows are created if the path of the default route is utilised. So I don't think this is a case of improper configuration.

$ mptcpize run iperf3 -c hostIPv4
[       CREATED] token=7921b290 remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52718 dport=5201
[   ESTABLISHED] token=7921b290 remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52718 dport=5201
Connecting to host hostIPv4, port 5201
[       CREATED] token=0b497c8d remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52728 dport=5201
[  5] local 192.168.1.2 port 52728 connected to hostIPv4 port 5201
[   ESTABLISHED] token=0b497c8d remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52728 dport=5201
[SF_ESTABLISHED] token=0b497c8d remid=0 locid=1 saddr4=192.168.13.2 daddr4=hostIPv4 sport=40195 dport=5201 backup=0 ifindex=11
[SF_ESTABLISHED] token=7921b290 remid=0 locid=1 saddr4=192.168.13.2 daddr4=hostIPv4 sport=35617 dport=5201 backup=0 ifindex=11
[SF_ESTABLISHED] token=7921b290 remid=0 locid=2 saddr4=192.168.14.2 daddr4=hostIPv4 sport=32779 dport=5201 backup=0 ifindex=12
[SF_ESTABLISHED] token=0b497c8d remid=0 locid=2 saddr4=192.168.14.2 daddr4=hostIPv4 sport=60581 dport=5201 backup=0 ifindex=12

The server only needs to use a single endpoint so it is a half mesh topology. To that extend, I've taken away the fullmesh flag on both subflows:

$ sudo ip mp e
192.168.13.2 id 1 subflow dev enp3s0.3 
192.168.14.2 id 2 subflow dev enp3s0.4

The behaviour is the same.

arinc9 commented 3 months ago

I think I'm onto something. I've been struggling with making MPTCP create subflows on a system without playing around with the routes. Limits are fine on both sides. I've noticed that subflows won't be created if the interface for the main flow is a subinterface, meaning it's a user of another interface. lan1@eth0, a DSA interface; enp3s0.3@enp3s0, a VLAN interface. I'll investigate further.

Edit: I've updated the main comment with my findings.

asirir commented 3 months ago

Tbh, I have a similar setup - multiple sub interfaces on the main interfaces (carrying different subnets). I don't have any vlan setup. I am able to establish subflows between just the sub-interfaces (with the primary flow also on the other sub-interface).

arinc9 commented 3 months ago

I don't use alias interfaces so I can't make a comment. There clearly is something wrong with MPTCP when using VLAN and DSA interfaces.

pabeni commented 3 months ago

Note that the mptcp monitor shows the lack of MPTCP_EVENT_ESTABLISHED events, thus the mptcp connection will not try to establish additional subflows.

Can you observe the same behavior on top of the current 'net' or 'net-next' tree?

The attached tentative reproducer works fine for me on a somewhat recent 'net-next'. repro2.sh.txt (I had to append a '.txt' suffix to please the github infra).

I guess 6.8 is lacking some fixes/improvement.

Could you also please attach a pcap captures with the first few packets of the established connection?

arinc9 commented 2 months ago

I will test this soon. Thanks Paolo.

matttbe commented 2 months ago

The default route is via 192.168.13.1 dev enp3s0.3.

I've set up the subflow I want to use:

$ sudo ip mp e
192.168.14.2 id 2 subflow dev enp3s0.4

Be careful that when you specify the device when adding the endpoint, the subflow will be forced to use this outgoing interface ...

Then described the gateway for the subflow path:

sudo ip rule add from 192.168.14.2 table 100
sudo ip route add default via 192.168.14.1 table 100

... In this case, the rule you added is not going to be used by the subflow I think, because oif is missing.

You can check your routing rules with ip route get <destIP> from <srcIP> oif <iface> (required IPRoute2, not the one from Busybox), e.g.

ip route get 1.1.1.1 from 192.168.14.2 oif enp3s0.4
matttbe commented 2 months ago

In other words, if you added the correct interface with the endpoint, you should not need to add these rules. That's what is mentioned in the 'warning' box: https://www.mptcp.dev/pm.html