exclude an interface from MPTCP

[VenkateswaranJ]

I have a client and server machine with three interfaces. I have configured source-based routing and enabled MPTCP on both sides. Now when I send a message between server and client, MPTCP automatically creates 3 subflows covering all three interfaces. For some reason, I don't want to create a subflow on particularly eth3, so is there any way to achieve this.

I'm using MPTCP version 0 and fullmesh as net.mptcp.mptcp_path_manager.

[matttbe]

Hi,

With the Fullmesh PM, you can used a modified version of IPRoute2 to flag some interfaces not to be used by the PM: http://multipath-tcp.org/pmwiki.php/Users/Tools (there is also a note not to use a modified version of IPRoute2)

You can see IP used by this PM in /proc/net/mptcp_fullmesh

[VenkateswaranJ]

Hi @matttbe,

Thanks for your comment. I can able to add and remove the interface by editing the flag as well as modified iproute2.

I have a few other requirements: a) Is it possible to get notified (either via POSIX API or any other way) when MPTCP fall back to TCP for some reason? b) Instead of using the fullmesh path manager, is it possible to start with default and add subflows when needed at runtime?

Is there any way to achieve this in MPTCP version 1 from the POSIX API level or some other way at userspace code? I would like to control subflow and get the status of the MPTCP connection (specifically getting some event when it falls back to TCP) at run time from userspace.

[VenkateswaranJ]

[Aeres-u99]

iirc, using ss in conjunction with iproute and ss, along with tcpdump. https://developers.redhat.com/blog/2020/08/19/multipath-tcp-on-red-hat-enterprise-linux-8-3-from-0-to-1-subflows/ ss will allow you to fetch detail of connection and with a simple script you can parse the output that will enable you to know when MPTCP has fallen back to TCP

For your second question I would suggest you to use progMP as it allows custom scheduler, and if you can somehow at runtime recognize the need to switch then you can on the fly change the subflow. I donot feel that this would be very effective, and is indeed a primitive solution. Still you can definitely try that out.

[matttbe]

a) Is it possible to get notified (either via POSIX API or any other way) when MPTCP fall back to TCP for some reason?

With this version of MPTCP (not the upstream one), you can do a getsockopt with MPTCP_ENABLED to know if a fallback has been done: http://multipath-tcp.org/pmwiki.php/Users/ConfigureMPTCP

You could also use ss (or inet diag) as suggested by @Aeres-u99 .

b) Instead of using the fullmesh path manager, is it possible to start with default and add subflows when needed at runtime?

Not with this version of MPTCP. The default path-manager only accepts new subflow: it doesn't initiate anything nor send addresses. If you want more flexibility, the best is to use the netlink path manager and control it from the userspace, e.g. with mptcpd.

Note that if you don't want subflows 3 and 4 on your schema, you can also add firewall (IPTables) or source routing rules to block them.

Is there any way to achieve this in MPTCP version 1 from the POSIX API level or some other way at userspace code?

What do you mean with "MPTCP version 1". There are currently two versions of the protocol: RFC6824 (v0) and RFC8684 (v1). The dev version of this MPTCP implementation (not the upstream version) supports both.

On our side at Tessares, we use the Netlink path manager we implemented with a userspace app controlling the different subflows. It allows us more control and avoid having to "block" the creation of some subflows. This is described in:

• https://dl.acm.org/doi/10.1145/2716281.2836113
• https://conferences2.sigcomm.org/co-next/2015/img/papers/conext15-final169.pdf

You could also tweak the scheduler but here, if you don't want to use a specific path, best not to have this path :)

Can we close this ticket?

[VenkateswaranJ]

@matttbe Thanks for your detailed comment and for this excellent pdf doc. Before closing this issue, It would be really helpful if you clearly few things.

Just to avoid unnecessary confusion I assume that your above answer only talks about the out-of-tree implementation of MPTCP (not the upstream implementation).

1. I read the pdf which you shared and would like to know where can I find the PM library according to the image mentioned in the pdf doc.

I assume the mptcp_netlink.c which you shared above is the implementation of Netlink pm on the kernel side? or shall I take mptcpd as an example userspace app that controls subflows via Netlink? Also, in the pdf, they mentioned that there is an event for the MPTCP connection created and closed.

The created event is triggered when a Multipath TCP connection is established. It contains the four-tuple, the id of the initial subflow and other information required to identify the connection. The estab event indicates the success of the three-way hand- shake and the closed event marks the termination of the Multipath TCP connection.

This closed event wouldn't be triggered when the MPTCP connection falls back to TCP? Because the MPTCP connection is closed and fall back to TCP.

2. In future, Out-of-tree implementation of RFC8684 (v1 in mptcp_trunk branch) will compatible with the upstream implementation of RFC8684 (v1) ?

[matttbe]

Just to avoid unnecessary confusion I assume that your above answer only talks about the out-of-tree implementation of MPTCP (not the upstream implementation).

Yes, that's correct. This kernel has a few names: out-of-tree, off-tree, mptcp.org, fork, etc. and linked to this Github repo: https://github.com/multipath-tcp/mptcp

I read the pdf which you shared and would like to know where can I find the PM library according to the image mentioned in the pdf doc.

Tessares has one but closed source. You can contact them if you are interested by that app with the support. mptcpd is an open-source alternative.

I assume the mptcp_netlink.c which you shared above is the implementation of Netlink pm on the kernel side?

Correct. This PM exposes a Netlink API for userspace apps (like the Tessares one or mptcpd) to control MPTCP subflows.

This closed event wouldn't be triggered when the MPTCP connection falls back to TCP? Because the MPTCP connection is closed and fall back to TCP.

I start to mix the two kernel implementation, we need to check. If you have a setup, you can turn on MPTCP debug (sysctl net.mptcp.mptcp_debug) → this event is fired when you this line is displayed in dmesg: mptcp_close: Close of meta_sk with tok (...). In mptcp.org kernel, you can still query the kernel using MPTCP_ENABLED sockopt.

In future, Out-of-tree implementation of RFC8684 (v1 in mptcp_trunk branch) will compatible with the upstream implementation of RFC8684 (v1) ?

Yes it should. For the moment, the focus is bigger on the upstream kernel. But the out-of-tree one is still maintained. Mainly by Tessares for its own business (and its clients)

[VenkateswaranJ]

@matttbe

I start to mix the two kernel implementation, we need to check. If you have a setup, you can turn on MPTCP debug (sysctl net.mptcp.mptcp_debug) → this event is fired when this line is displayed in dmesg: mptcp_close: Close of meta_sk with tok (...).

To make my intention clear, I'm looking for a Netlink event for TCP fallback because I don't want to poll every 1 sec or something with getsockopt, but this is only needed if TCP fallback happens in the middle of the client-server communications.

Does TCP fallback always happen at the connection establishment stage? (either version incompatible or due to middle box)

I tried with my test setup. cables are directly connected between sever and client PCs and there is no source routing since it's a direct connection. eth0(10.10.10.XX/24) and eth1(10.10.20.XX/24). server command: nc -l ip1 port1 client command: nc -s ip1 serverip1 serverport1

But I could not create the TCP fallback scenario in the middle of the client-server communication. I have tried the below things.

1. multipath turned off via ip tool on both interface at the client side at run time and expect the MPTCP connection will be fall back to TCP but instead it terminates the connection completely . Is it the right behavior?

But when the whole connection terminates I got the below event in dmesg. +1,398830] mptcp_close: Close of meta_sk with tok 0x3740c986

2. multipath turned off on both interfaces on the client-side before starting the application but this time client starts a normal TCP connection [syn, syn_ack, ack] this makes sense.

3. multipath turn off only works on the client-side, for example, if I turn on multipath on both interfaces at client-side and turn off multipath at server side for both interface then I expect the server to reject the MP_CAPABLE request from client and start the normal TCP, but instead server responds for that request and initiate the MPTCP connection. why does the server accept mp_capable request from the client even though I turned off mptcp on both of its interfaces?

4. Turned off mptcp at the system level on the server-side (sysctl -w net.mptcp.mptcp_enabled=0). This time client sends MP_CAPABLE syn to the server but server silently ignores and start normal TCP connection and there is no dmesg log at both side.

[matttbe]

Does TCP fallback always happen at the connection establishment stage? (either version incompatible or due to middle box)

I don't have numbers but most of the time, it should happen at the connection establishment (MPTCP_EVENT_ESTABLISHED). If it happens later, it means a middlebox has altered an ongoing connection. I guess that should not be common.

multipath turned off via ip tool on both interface at the client side at run time and expect the MPTCP connection will be fall back to TCP but instead it terminates the connection completely . Is it the right behavior?

The fullmesh PM will remove subflow(s) using the IP(s) linked to the interface you just set off. The goal is not to fallback to TCP.

why does the server accept mp_capable request from the client even though I turned off mptcp on both of its interfaces?

Due to the architecture, if the socket has MPTCP capabilities, it will accept the MPTCP connection. If you don't want to accept MPTCP on a socket, you should instead not create a socket with MPTCP capabilities. For that, you can turn off MPTCP with sysctl -w net.mptcp.mptcp_enabled=0 or set it to 2 and add the MPTCP_ENABLED socket option.

Turned off mptcp at the system level on the server-side (sysctl -w net.mptcp.mptcp_enabled=0). This time client sends MP_CAPABLE syn to the server but server silently ignores and start normal TCP connection and there is no dmesg log at both side.

I guess you can be notified (select, poll, epoll, etc.) when you are connected and use getsockopt(..., MPTCP_ENABLED,...) to check the status. Would that be possible for you?

[VenkateswaranJ]

I guess you can be notified (select, poll, epoll, etc.) when you are connected and use getsockopt(..., MPTCP_ENABLED,...) to check the status. Would that be possible for you?

@matttbe, As you suggested I have tested with simple c code by turning off mptcp at server-side mptcp_enabled=0 and try to connect to it from mptcp capable client and the MPTCP_ENABLED opt set to 0 at client-side indicating that the connection is not mptcp. It works as I expected.

Sorry for asking multiple questions on this same issue, but one last thing. Do you know any way to simulate the TCP fallback in the middle of the client-server communication? (any networking tool or something ?)

Because the above test only simulates the TCP fallback at the initial stage, if the fallback happened in the middle of the connection the I need to figure out a way to detect this (checking getsockopt() with a regular interval ?)

If you don't have an idea for stimulating TCP fallback in the middle of the connection then please close this issue.

Once again a big thank you for supporting people on mptcp Github issues. Really appreciate your time!

[matttbe]

As you suggested I have tested with simple c code by turning off mptcp at server-side mptcp_enabled=0 and try to connect to it from mptcp capable client and the MPTCP_ENABLED opt set to 0 at client-side indicating that the connection is not mptcp. It works as I expected.

:+1:

Sorry for asking multiple questions on this same issue, but one last thing. Do you know any way to simulate the TCP fallback in the middle of the client-server communication? (any networking tool or something ?)

Maybe by modifying MPTCP option and re-compute the TCP checksum? Maybe with Scapy?

Because the above test only simulates the TCP fallback at the initial stage, if the fallback happened in the middle of the connection the I need to figure out a way to detect this (checking getsockopt() with a regular interval ?)

Yes, I guess you can. You can also monitor MPTCP SNMP counters in /proc/net/mptcp_net/snmp or propose a patch to send a Netlink event. But again, I think that should be rare.

As you said, we can close this issue as the original question has been been replied.