Restoration subflow after the breaks

[LovingFox]

I try to test mptcp (v0.94.2 and linux 4.14.98) and found that the subflows are not restored after the break.

Schema of the lab:

  ┌wan1(ip1)─────┐
cpe             (ip) Server
  └wan2(ip2)─────┘

At first CPE connects to the Server (ip) and creates two subflows through wan1 (ip1) and wan2 (ip2): ip1->ip and ip2->ip. So far so good.

Next, I set tcp-filter on the way wan1->Server and subflow ip1->ip is breaking down:

  ┌wan1(ip1)───x─┐
cpe             (ip) Server
  └wan2(ip2)─────┘

Here everything is predictable.

Next, I remove tcp-filter on the way wan1->Server. But subflow ip1->ip does not recover... Is it rigth behovior for the MPTCP? Is it possible to configure the MPTCP so that it restores all flows after a break?

Configuration:

root@OpenWrt:/# sysctl net.mptcp
net.mptcp.mptcp_binder_gateways = 
net.mptcp.mptcp_checksum = 0
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0

root@OpenWrt:/# cat /sys/module/mptcp_fullmesh/parameters/create_on_err
1

[knxhm]

I had the same observation and the explanation of Christoph was , if mptcp receives an icmp unreachable then the interrupted subflow does not restart. Only new sessions would use this connection again. Even if create_on_err is set to 1. Am 13.02.19, 15:43, Ivan Revyakin notifications@github.com schrieb: I try to test mptcp (v0.94.2 and linux 4.14.98) and found that the subflows are not restored after the break.

Schema of the lab:

┌wan1(ip1)─────┐

cpe (ip) Server

└wan2(ip2)─────┘

At first CPE connects to the Server (ip) and creates two subflows through wan1 (ip1) and wan2 (ip2): ip1->ip and ip2->ip. So far so good.

Next, I set tcp-filter on the way wan1->Server and subflow ip1->ip is breaking down:

┌wan1(ip1)───x─┐

cpe (ip) Server

└wan2(ip2)─────┘

Here everything is predictable.

Next, I remove tcp-filter on the way wan1->Server. But subflow ip1->ip does not recover...

Is it rigth behovior for the MPTCP?

Is it possible to configure the MPTCP so that it restores all flows after a break?

Configuration:

root@OpenWrt:/# sysctl net.mptcp

net.mptcp.mptcp_binder_gateways =

net.mptcp.mptcp_checksum = 0

net.mptcp.mptcp_debug = 0

net.mptcp.mptcp_enabled = 1

net.mptcp.mptcp_path_manager = fullmesh

net.mptcp.mptcp_scheduler = default

net.mptcp.mptcp_syn_retries = 3

net.mptcp.mptcp_version = 0

root@OpenWrt:/# cat /sys/module/mptcp_fullmesh/parameters/create_on_err

1

—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/multipath-tcp/mptcp","title":"multipath-tcp/mptcp","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/multipath-tcp/mptcp"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Restoration subflow after the breaks (#316)"}],"action":{"name":"View Issue","url":"https://github.com/multipath-tcp/mptcp/issues/316"}}}

[

{

"@context": "http://schema.org",

"@type": "EmailMessage",

"potentialAction": {

"@type": "ViewAction",

"target": "https://github.com/multipath-tcp/mptcp/issues/316",

"url": "https://github.com/multipath-tcp/mptcp/issues/316",

"name": "View Issue"

},

"description": "View this Issue on GitHub",

"publisher": {

"@type": "Organization",

"name": "GitHub",

"url": "https://github.com"

}

}

]

[matttbe]

Hi,

Is it possible to configure the MPTCP so that it restores all flows after a break?

Currently, if you set create_on_err param, it will only re-create the SF in case of timeout: https://github.com/multipath-tcp/mptcp/blob/mptcp_v0.94/net/mptcp/mptcp_fullmesh.c#L1445 (mainly to recover in case of NAT-related issues). (Note that this code has changed in mptcp_trunk branch: https://github.com/multipath-tcp/mptcp/commit/ed3036438443a7bcd8d2d0169a012d5595baed75 + the parent)

How does your filter work?

[LovingFox]

matttbe: How does your filter work?

At Server side (ip1 is a wan1 address of cpe):

sudo iptables -I OUTPUT 1 -j DROP -d <ip1>
sudo iptables -I INPUT 1 -j DROP -s <ip1>

matttbe: Currently, if you set create_on_err param, it will only re-create the SF in case of timeout: knxhm: I had the same observation and the explanation of Christoph was , if mptcp receives an icmp unreachable then the interrupted subflow does not restart.

As you see above the iptables filter will not produce any icmp (icmp-unreachable including) due to DROP target.

Anyway, what can be done to make an established MPTCP session try to produce new subflows permanently?

For example. I create a vpn tunnel using MPTCP as a transport and I hope this tunnel will use all possible ways to communicate Sever all mptcp-session life. Tunnel established and uses wan1 and wan2 interfaces. After a while the way via wan1 dropped for some reason and subflow ip1->ip is broken. A more bit after that the way is available again. But the vpn-application will never know about it and will not try re-establish a subflow via wan1 while the flow through the wan2 work.

[matttbe]

Anyway, what can be done to make an established MPTCP session try to produce new subflows permanently?

This feature of the fullmesh path manager is defined like this:

• if a subflow is closed due to an error, the PM will be notified
• if this error is ETIMEDOUT, create_on_err is different than 0 and the linekd MPTCP connection still alive, then a recreation of this subflow will be scheduled.

Note that:

• with mptcp_v0.94, the recreation of this subflow might be scheduled too soon and fail, see the modifications in mptcp_trunk
• the timeout can be very long. If I remember, ETIMEDOUT can be very long and will depend of different parameters (e.g. tcp_retries2) but by default, you would need something like ~15 minutes of inactivity.
• if after the removal of your IPTables rules, the network changes and, for some reasons, the subflow is terminated with another error than ETIMEDOUT (no route, RST received, interface down, etc.), the PM will not retry to establish a new subflow. Feel free to hook full_mesh_subflow_error() and print the value of sk->sk_err ;-)

[cpaasch]

That's an opportunity for a netlink-based path-manager IMO!

mptcp_trunk exposes a new API for such kind of a path-manager.

[LovingFox]

matttbe: Feel free to hook full_mesh_subflow_error() and print the value of sk->sk_err ;-)

When I set the iptables-filter the value of sk->sk_err is 148: #define EHOSTUNREACH 148 /* No route to host */

Is this a good idea to add a check sk->sk_err != EHOSTUNREACH to the full_mesh_subflow_error?

if (sk->sk_err != ETIMEDOUT && sk->sk_err != EHOSTUNREACH)
        return;

cpaasch: That's an opportunity for a netlink-based path-manager IMO!

Unfortunately I don't have so much knowledge and experience to make this path-manager...

[matttbe]

Is this a good idea to add a check sk->sk_err != EHOSTUNREACH to the full_mesh_subflow_error?

Not sure, not for a generic path-manager. It means that it doesn't make sense to try to establish the new subflow. We could re-schedule with a delay but unfortunately, that depends of the use-case.

@cpaasch : what do you think?

[cpaasch]

Is this a good idea to add a check sk->sk_err != EHOSTUNREACH to the full_mesh_subflow_error?

Not sure, not for a generic path-manager. It means that it doesn't make sense to try to establish the new subflow. We could re-schedule with a delay but unfortunately, that depends of the use-case.

@cpaasch : what do you think?

Yeah, the risk is that we end up "looping". We might try to do some tricks with exponential backoff,...

Even if it's hard to do, the right way forward should be a netlink-based path-manager that can monitor the state of the routes/paths.

[LovingFox]

Ok guys. As I understood a good way to solve the issue is:

1. to make netlink-based path-manager
2. do it using new API of mptcp_trunk

Is it wright?

[matttbe]

I think that's the cleanest way for your use case.

[LovingFox]

Could you tell me are there some kind of documentation for API mentioned above that may helps to create new path-manager?

[matttbe]

Unfortunately, the doc is currently only in the code: https://github.com/multipath-tcp/mptcp/blob/mptcp_trunk/include/uapi/linux/mptcp.h#L49 (but anybody can add this doc of course :) )

I know that some people at Intel are working on Open-Sourcing a userspace PM using Netlink.

Someone is also looking at having a userspace PM in haskell: https://github.com/multipath-tcp/mptcp/issues/308#issuecomment-456274636

Of course, for your use case, it would be quicker to modify the C code on your own branch but you might be interested at also tweaking this PM more to your use-case. In this case, looking at the Netlink API is the way to go!

I hope you don't mind if I close this ticket!