What is the address id of main subflow

[VenkateswaranJ]

When the ip address of the subflow changes on any side I can send MPTCP_CMD_REMOVE with token and local address id and it will remove the subflow and also inform the remote that this IP address is no more available.

But how can I do this on the main subflow? because it's created by TCP client/server application and I didn't set any address id for those IPs. I have seen somewhere in the Github issue that says that address id 0 is a special one and is used for the main subflow so I tried to send the MPTCP_CMD_REMOVE command with address id 0 but it close the whole MPTCP connection instead of closing only the main subflow.

How can I handle IP address change on the main subflow? I would Like to do the below things

1. when the IP address changes, remove the subflow associated with that ip.
2. create a new subflow with new IP without closing MPTCP connection.

Note: MPTCP_CMD_SUB_DESTROY command does not close the subflow TCP connection.

[matttbe]

Hello

I have seen somewhere in the Github issue that says that address id 0 is a special one and is used for the main subflow

Yes, that's correct.

so I tried to send the MPTCP_CMD_REMOVE command with address id 0 but it close the whole MPTCP connection instead of closing only the main subflow.

What is the PM on the host which received the RM_ADDR? Please note that the kernel doesn't check if the userspace gives the right address ID. If I'm not mistaken, when the userspace PM creates a subflow, it can set the local ID 0 for example even if the source IP is different from the original IP. The kernel doesn't check that because the userspace could have more info or really wants to do that for some reasons.

So it looks like there is a bug somewhere: either in the PM managing the remote IDs or wrong info have been shared. Did you check the IDs set to the MP_JOIN in your example? Did you try by setting the fullmesh PM on host receiving the RM_ADDR?

How can I handle IP address change on the main subflow?

I would to the item 2 then 1: create the new subflow then remove the previous one → just in case it was the last subflow of the MPTCP connection and the other host decides to destroy MPTCP connections without subflows.

MPTCP_CMD_SUB_DESTROY command does not close the subflow TCP connection.

Strange, I'm pretty sure it is working on my side. Do you have an error returned by the kernel? Maybe best to open a dedicated issue for that. Except if you think it is linked to this issue.

[VenkateswaranJ]

@matttbe apologize for the late response and also for the little confusion in my above question. This is probably not a bug because I found that when I'm sending MPTCP_CMD_REMOVE on address id 0 it removes the main subflows but at the same time the second subflow network had some issue so it might be the reason it closes the whole connection since there is no valid subflow to proceed further. I have fixed the issue in the second subflow network and tested it again and it works as expected.

What is the PM on the host which received the RM_ADDR?

net.mptcp.mptcp_path_manager = netlink I'm using Netlink path manager on both the server and client sides.

Please note that the kernel doesn't check if the userspace gives the right address ID. If I'm not mistaken, when the userspace PM creates a subflow, it can set the local ID 0 for example even if the source IP is different from the original IP. The kernel doesn't check that because the userspace could have more info or really wants to do that for some reason.

yes, you are right! I tested this.

It would be great if you also clarify the below questions.

1. MPTCP_EVENT_ESTABLISHED

   New subflows can only be created after the mptcp connection has been established and to establish the mptcp connection, the server and client have to perform some data transfer. If the server and client are only connected and no further data is transferred between them then the mptcp connection will never be established. am I right here?

2. MPTCP_CMD_REMOVE

   When I send remove address ID to a remote host, it destroys the subflow and closes the TCP connection. Is it the right behaviour?

3. MPTCP_CMD_ANNOUNCE and MPTCP_CMD_REMOVE

   When a local host sends any of these messages to a remote host, is it guaranteed that this message will be delivered until a valid mptcp connection exists between them or should the local host needs to send it periodically until the remote host receives it?

4. Handling Address Change & and Link down

   MPTCP_CMD_REMOVE and MPTCP_CMD_SUB_DESTROY commands work fine if I execute them on an active subflow connection. It closes the TCP connection of a particular subflow and also removes the mptcp subflow socket on both sides. The problem is, it doesn't work on link down and address change scenarios, it seems obvious that when the link goes down (or IP address removed from the interface) local host sends MPTCP_CMD_REMOVE to inform the remote that this IP is no more exist. Now the remote host has to close the TCP connection and remove the mptcp subflow socket, But it can't send TCP RST as the destination IP did not exist anymore so it only clears the subflow socket locally. On the other side (the host that sends the remove IP command) mptcp subflow socket is not removed also there is no MPTCP_EVENT_SUB_CLOSED Netlink event. To destroy this Zombie subflow socket I need to call MPTCP_CMD_SUB_DESTROY even though the subflow has been removed on the remote side (but didn't update the other side in some way).

5. FullMesh Path manager

   fullmesh path manager handles both link down and address change scenarios by monitoring IP address changes on all interfaces, but I wonder how it removes the mptcp subflow socket on both ends? for example, when an IP address is removed from an Interface it sends MPTCP_CMD_REMOVE to the remote host but at the same time, it removes the subflow socket locally. I can understand that it has more power since it's sitting in kernel space but is there any way I can achieve this in the Netlink path manager? one way would be calling the MPTCP_CMD_SUB_DESTROY command on both sides, but is it the right thing to do?

6. Do you have any idea, how many times the full-mesh path manager retries to establish subflow before it gives up.

7. Is there any /proc file location where I can get the total subflow count of the mptcp connection?

   $ cat /proc/net/mptcp_net/mptcp sl | loc_tok rem_tok | v6 | local_address | remote_address | st | ns | tx_queue rx_queue | inode

What's the meaning of ns in the above proc file? it's matching with the subflow count.

[matttbe]

Sorry, I was mostly off the past few days and a lot to catchup!

If the server and client are only connected and no further data is transferred between them then the mptcp connection will never be established. am I right here?

Yes, that's correct. When the first bytes have been transferred with the correct MPTCP options, the state switch to "fully established".

When I send remove address ID to a remote host, it destroys the subflow and closes the TCP connection. Is it the right behaviour?

Yes, that's the expected one

When a local host sends any of these messages to a remote host, is it guaranteed that this message will be delivered until a valid mptcp connection exists between them or should the local host needs to send it periodically until the remote host receives it?

With MPTCPv0 (RFC6824), ADD_ADDR are not retransmitted if they are lost and linked to that, there is no "ACK" to know it has been transmitted. You might have to re-send it later if no new subflow is established.

Side note: technically, you can send the ADD_ADDR before MPTCP_EVENT_ESTABLISHED. It depends if the receivers can interpret them but a Netlink PM will forward the event to the userspace.

On the other side (the host that sends the remove IP command) mptcp subflow socket is not removed also there is no MPTCP_EVENT_SUB_CLOSED Netlink event. To destroy this Zombie subflow socket I need to call MPTCP_CMD_SUB_DESTROY even though the subflow has been removed on the remote side (but didn't update the other side in some way).

Good point. I think that it is the userspace PM that is the owner of the connection and has to close the subflow in this case. I see that the in-kernel doesn't call mptcp_sub_force_close() except if you can the SUB_DESTROY command.

one way would be calling the MPTCP_CMD_SUB_DESTROY command on both sides, but is it the right thing to do?

Yes it is

Do you have any idea, how many times the full-mesh path manager retries to establish subflow before it gives up.

After a very quick look, I see it removes the bitfield just after. But I didn't check if it could re-add it later if it fails. Hint 1

What's the meaning of ns in the above proc file? it's matching with the subflow count.

Yes, ns is what you are looking for: Number of Subflows

[VenkateswaranJ]

@matttbe Thanks again! Shall we close this issue?

[matttbe]

Sure, thank you for the reminder!