Problem using MPTCP version 1

[praupadhyaya]

Hi,

I am using MPTCP version 0.95.1. I am trying to interwork between a custom MPTCP Stack and Linux running MPTCP 0.95.1 With MPTCP version 0, everything is fine. When I use MPTCP version 1, I run into the problem that Linux MPTCP is falling back to TCP. My custom stack side (Host-A) is client, Linux with MPTCP version 0.95.1 is Host-B. As per the RFC 8684, the following handshake takes place for the first master connection (usecase is Host-A has data later to send, in my case an HTTP get with wget) -- I quote from RFC 8684 Host A (with data to send later) Host B

---

  MP_CAPABLE                ->
  [flags]
                            <-            MP_CAPABLE
                                          [B's key, flags]
  ACK + MP_CAPABLE          ->
  [A's key, B's key, flags]

  ACK + MP_CAPABLE + data   ->
  [A's key, B's key, flags, data-level details]

After the Host-A sends the data (note that RFC is not obligating in the first data to send the DSS), I see that Host-B does respond with the HTTP response but falls back to TCP instead of responding with MPTCP with a DSS. Is there a bug that because DSS is not present in the first data from Host-A (as I said, not mandated by RFC as per scenario above), Linux stack is falsely assuming that the path is not MPTCP capable ? pcap attached. Set the following filter (ip.src == 11.11.11.1 and ip.dst == 10.3.10.165) or (ip.src == 10.3.10.165 and ip.dst == 11.11.11.1) Host-A = 11.11.11.1 and Host-B = 10.3.10.165 (Linux running MPTCP v0.95.1) N6_mptcp_pdn_http_31.zip

Regards -Prashant

[matttbe]

Hi Prashant,

MPTCP 0.95.1 doesn't fully support MPTCP version 1. In short, it doesn't support the new way of initiating an MPTCP connection. The 0.95 version has been released while RFC 8684 was still in progress.

If you are interested by MPTCP version 1, you can either build the development version by using the mptcp_trunk branch or switch to the upstream kernel, e.g. the 5.12 kernel version.

Note that we are interested by interop results, especially for the MPTCP Upstream project. By chance, can you tell us more about your custom stack? Could you eventually share the status? Do not hesitate to attend MPTCP Upstream weekly meetings -- with people from Intel, Apple, Tessares, RedHat, Xiaomi and ChinaTelecom -- if you are looking at the interop point of view.

Cheers, Matt

[sdntechforum]

Hi,

I am using MPTCP version 0.95.1. I am trying to interwork between a custom MPTCP Stack and Linux running MPTCP 0.95.1 With MPTCP version 0, everything is fine. When I use MPTCP version 1, I run into the problem that Linux MPTCP is falling back to TCP. My custom stack side (Host-A) is client, Linux with MPTCP version 0.95.1 is Host-B. As per the RFC 8684, the following handshake takes place for the first master connection (usecase is Host-A has data later to send, in my case an HTTP get with wget) -- I quote from RFC 8684 Host A (with data to send later) Host B

---

MP_CAPABLE -> [flags] <- MP_CAPABLE [B's key, flags] ACK + MP_CAPABLE -> [A's key, B's key, flags]

  ACK + MP_CAPABLE + data   ->
  [A's key, B's key, flags, data-level details]

After the Host-A sends the data (note that RFC is not obligating in the first data to send the DSS), I see that Host-B does respond with the HTTP response but falls back to TCP instead of responding with MPTCP with a DSS. Is there a bug that because DSS is not present in the first data from Host-A (as I said, not mandated by RFC as per scenario above), Linux stack is falsely assuming that the path is not MPTCP capable ? pcap attached. Set the following filter (ip.src == 11.11.11.1 and ip.dst == 10.3.10.165) or (ip.src == 10.3.10.165 and ip.dst == 11.11.11.1) Host-A = 11.11.11.1 and Host-B = 10.3.10.165 (Linux running MPTCP v0.95.1) N6_mptcp_pdn_http_31.zip

Regards -Prashant

Hey Prashant,

How can I DM you, looking for some guidance on installation, looks like you got it woring. Thanks, Amit

[praupadhyaya]

Hi Amit,

You can reach out to me at @.***

Regards -Prashant

On Thu, Jun 17, 2021 at 1:33 AM sdntechforum @.***> wrote:

Hi,

I am using MPTCP version 0.95.1. I am trying to interwork between a custom MPTCP Stack and Linux running MPTCP 0.95.1 With MPTCP version 0, everything is fine. When I use MPTCP version 1, I run into the problem that Linux MPTCP is falling back to TCP. My custom stack side (Host-A) is client, Linux with MPTCP version 0.95.1 is Host-B. As per the RFC 8684, the following handshake takes place for the first master connection (usecase is Host-A has data later to send, in my case an HTTP get with wget) -- I quote from RFC 8684 Host A (with data to send later) Host B

MP_CAPABLE -> [flags] <- MP_CAPABLE [B's key, flags] ACK + MP_CAPABLE -> [A's key, B's key, flags]

ACK + MP_CAPABLE + data -> [A's key, B's key, flags, data-level details]

After the Host-A sends the data (note that RFC is not obligating in the first data to send the DSS), I see that Host-B does respond with the HTTP response but falls back to TCP instead of responding with MPTCP with a DSS. Is there a bug that because DSS is not present in the first data from Host-A (as I said, not mandated by RFC as per scenario above), Linux stack is falsely assuming that the path is not MPTCP capable ? pcap attached. Set the following filter (ip.src == 11.11.11.1 and ip.dst == 10.3.10.165) or (ip.src == 10.3.10.165 and ip.dst == 11.11.11.1) Host-A = 11.11.11.1 and Host-B = 10.3.10.165 (Linux running MPTCP v0.95.1) N6_mptcp_pdn_http_31.zip https://github.com/multipath-tcp/mptcp/files/6604046/N6_mptcp_pdn_http_31.zip

Regards -Prashant

Hey Prashant,

How can I DM you, looking for some guidance on installation, looks like you got it woring. Thanks, Amit

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-862675356, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXWGRQLT34ITZI3GD3TTTD7SNANCNFSM46FZEKNA .

[sdntechforum]

Hi Amit, You can reach out to me at @. Regards -Prashant … On Thu, Jun 17, 2021 at 1:33 AM sdntechforum @.> wrote: Hi, I am using MPTCP version 0.95.1. I am trying to interwork between a custom MPTCP Stack and Linux running MPTCP 0.95.1 With MPTCP version 0, everything is fine. When I use MPTCP version 1, I run into the problem that Linux MPTCP is falling back to TCP. My custom stack side (Host-A) is client, Linux with MPTCP version 0.95.1 is Host-B. As per the RFC 8684, the following handshake takes place for the first master connection (usecase is Host-A has data later to send, in my case an HTTP get with wget) -- I quote from RFC 8684 Host A (with data to send later) Host B ------------------------------ MP_CAPABLE -> [flags] <- MP_CAPABLE [B's key, flags] ACK + MP_CAPABLE -> [A's key, B's key, flags] ACK + MP_CAPABLE + data -> [A's key, B's key, flags, data-level details] After the Host-A sends the data (note that RFC is not obligating in the first data to send the DSS), I see that Host-B does respond with the HTTP response but falls back to TCP instead of responding with MPTCP with a DSS. Is there a bug that because DSS is not present in the first data from Host-A (as I said, not mandated by RFC as per scenario above), Linux stack is falsely assuming that the path is not MPTCP capable ? pcap attached. Set the following filter (ip.src == 11.11.11.1 and ip.dst == 10.3.10.165) or (ip.src == 10.3.10.165 and ip.dst == 11.11.11.1) Host-A = 11.11.11.1 and Host-B = 10.3.10.165 (Linux running MPTCP v0.95.1) N6_mptcp_pdn_http_31.zip https://github.com/multipath-tcp/mptcp/files/6604046/N6_mptcp_pdn_http_31.zip Regards -Prashant Hey Prashant, How can I DM you, looking for some guidance on installation, looks like you got it woring. Thanks, Amit — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#427 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXWGRQLT34ITZI3GD3TTTD7SNANCNFSM46FZEKNA .

I am sorry but I do not see the email address you shared, Prashant, can you please DM me at sdntechforum@gmail.com

[sdntechforum]

reply to Prashant

[praupadhyaya]

Hi Matt,

I did two experiments. First, I built from the head of mptcp_v0.95 This kernel did better -- with my own client (using MPTCPv1), the 3 way handshake on MPTCP succeeded (new kernel was running the server with MPTCPv1, this is progress on top of the problem I reported) However, when my client issued the HTTP GET on MPTCP, the kernel side responded with data but over TCP and not MPTCP, so this was a problem.

Second, I built from the head of mptcp_trunk, in this case the 3 way handshake itself falls back to TCP. My Client sent an MPTCP SYN, but the server sent a TCP SYN ACK, so the connection works end to end but is running TCP now instead of MPTCP.

In addition, I ran a wget running on the same machine as the new kernel (so none of my code), there too I see that handshake suddenly starts using TCP. So something looks amiss.

In general, I think MPTCPv1 is simply not ready on any of the branches. Has anyone used MPTCPv1 properly end to end at all ?

Regards -Prashant

On Mon, Jun 7, 2021 at 12:42 PM Matthieu Baerts @.***> wrote:

Hi Prashant,

MPTCP 0.95.1 doesn't fully support MPTCP version 1. In short, it doesn't support the new way of initiating an MPTCP connection. The 0.95 version has been released while RFC 8684 was still in progress.

If you are interested by MPTCP version 1, you can either build https://multipath-tcp.org/pmwiki.php/Users/DoItYourself the development version by using the mptcp_trunk branch or switch to the upstream kernel https://github.com/multipath-tcp/mptcp_net-next/wiki, e.g. the 5.12 kernel version.

Note that we are interested by interop results, especially for the MPTCP Upstream project. By chance, can you tell us more about your custom stack? Could you eventually share the status? Do not hesitate to attend MPTCP Upstream weekly meetings https://github.com/multipath-tcp/mptcp_net-next/wiki/Meetings -- with people from Intel, Apple, Tessares, RedHat, Xiaomi and ChinaTelecom -- if you are looking at the interop point of view.

Cheers, Matt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-855658514, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXVUHD764Q2JAFCT7E3TRRWOPANCNFSM46FZEKNA .

[matttbe]

Hi Prashant,

For mptcp_trunk, we only did validations between two identical kernel. Because of that, the support is still in progress.

We (Mat) also did some tests on MPTCP Upstream side and we discovered that the current mptcp_trunk version has some issues when there is data in the 4th ACK (MP_CAPABLE + DATA): https://lore.kernel.org/mptcp/4f24544d-d1f1-82f9-ce10-f5ea3806d137@tessares.net/T/#u

There is also an issue with the checksum support in mptcp_trunk: https://github.com/multipath-tcp/mptcp_net-next/issues/205

It looks like today, the reference implementation for MPTCPv1 is the "upstream" kernel, e.g. v5.12 version: https://github.com/multipath-tcp/mptcp_net-next/wiki Could you use this one?

If you do, may you share the status? For the upstream project, we are looking for interop with other stacks. Do not hesitate to reach us on our ML or other channels: https://github.com/multipath-tcp/mptcp_net-next/wiki

Cheers, Matt

[mjmartineau]

Here's some further information on the MPTCPv1 problems I encountered with mptcp_trunk and the upstream kernel:

1. There's an error in the definition of MPTCPV1_SUB_LEN_CAPABLE_DATA_CSUM. It should be 24 not 22. I am sending a patch for that.
2. mptcp_write_mpcapable_data() does a memcpy() that puts the DSS header bytes in to an MP_CAPABLE+DATA header. This is not the header format defined in RFC 8684, and corrupts the TCP options (can see this in the pcap I attached to multipath-tcp/mptcp_net-next#205)
3. On the receive side, mptcp_parse_options() and related functions appear to assume DSS-formatted header bytes in the MP_CAPABLE+DATA packets. When these (incorrect) headers are not found, mptcp_trunk falls back to TCP.

MP_CAPABLE+DATA headers are sent whenever the sender of the SYN also sends the first data, so the conditions in item 3 match up with @praupadhyaya's observations about a client sending HTTP GET.

[praupadhyaya]

Just to give a quick update here. With upstream kernel 5.13, I was able to interwork on MPTCPv1 with my own stack client. However I could do this only with MPTCP checksums disabled at the kernel MPTCPv1. I have not been able to get the kernel to agree with my generated checksums in packets so kernel sends an MPTCP RST.

Is anybody aware of any checksum related issue in the upstream kernel implementation, would appreciate any input, since I cannot optically find any issue in my own client code.

Regards -Prashant

On Fri, Jun 18, 2021 at 6:58 PM Matthieu Baerts @.***> wrote:

Hi Prashant,

For mptcp_trunk, we only did validations between two identical kernel. Because of that, the support is still in progress.

We (Mat) also did some tests on MPTCP Upstream side and we discovered that the current mptcp_trunk version has some issues when there is data in the 4th ACK (MP_CAPABLE + DATA): @.***/T/#u

There is also an issue with the checksum support in mptcp_trunk: multipath-tcp/mptcp_net-next#205 https://github.com/multipath-tcp/mptcp_net-next/issues/205

It looks like today, the reference implementation for MPTCPv1 is the "upstream" kernel, e.g. v5.12 version: https://github.com/multipath-tcp/mptcp_net-next/wiki Could you use this one?

If you do, may you share the status? For the upstream project, we are looking for interop with other stacks. Do not hesitate to reach us on our ML or other channels: https://github.com/multipath-tcp/mptcp_net-next/wiki

Cheers, Matt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-864039779, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXXPBYJ4LPBEHIAKBRDTTNCYPANCNFSM46FZEKNA .

[praupadhyaya]

Another observation with MPTCPv1 with upstream kernel. In this the socket needs to be created explicitly with the MPTCP protocol (wrapper script helps one do that) However, I am seeing that on this socket the IP_TRANSPARENT option cannot be set with a call like this (throws an error that operation is not permitted) -- setsockopt(s, SOL_IP, IP_TRANSPARENT, &opt, sizeof(opt)))

So looks like the above needs a fix.

Regards -Prashant

On Sat, Jul 3, 2021 at 12:28 AM Prashant Upadhyaya @.***> wrote:

Just to give a quick update here. With upstream kernel 5.13, I was able to interwork on MPTCPv1 with my own stack client. However I could do this only with MPTCP checksums disabled at the kernel MPTCPv1. I have not been able to get the kernel to agree with my generated checksums in packets so kernel sends an MPTCP RST.

Is anybody aware of any checksum related issue in the upstream kernel implementation, would appreciate any input, since I cannot optically find any issue in my own client code.

Regards -Prashant

On Fri, Jun 18, 2021 at 6:58 PM Matthieu Baerts @.***> wrote:

Hi Prashant,

For mptcp_trunk, we only did validations between two identical kernel. Because of that, the support is still in progress.

We (Mat) also did some tests on MPTCP Upstream side and we discovered that the current mptcp_trunk version has some issues when there is data in the 4th ACK (MP_CAPABLE + DATA): @.***/T/#u

There is also an issue with the checksum support in mptcp_trunk: multipath-tcp/mptcp_net-next#205 https://github.com/multipath-tcp/mptcp_net-next/issues/205

It looks like today, the reference implementation for MPTCPv1 is the "upstream" kernel, e.g. v5.12 version: https://github.com/multipath-tcp/mptcp_net-next/wiki Could you use this one?

If you do, may you share the status? For the upstream project, we are looking for interop with other stacks. Do not hesitate to reach us on our ML or other channels: https://github.com/multipath-tcp/mptcp_net-next/wiki

Cheers, Matt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-864039779, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXXPBYJ4LPBEHIAKBRDTTNCYPANCNFSM46FZEKNA .

[mjmartineau]

@praupadhyaya, is your own MPTCPv1 stack on a big-endian or little-endian system? Also, checksum support was added after kernel v5.13 (now it is in v5.14-rc1), were you using the net-next branch with for these checksum tests?

I'm wondering if we have any checksum endianness bugs lurking in the upstream code.

[praupadhyaya]

Hi Mat,

I was testing with 5.13-rc6. Will try the 5.14-rc1 as you indicated and will let you know the results. My MPTCPv1 stack is currently running on little-endian system (x86)

Regards -Prashant

On Thu, Jul 15, 2021 at 9:33 PM Mat Martineau @.***> wrote:

@praupadhyaya https://github.com/praupadhyaya, is your own MPTCPv1 stack on a big-endian or little-endian system? Also, checksum support was added after kernel v5.13 (now it is in v5.14-rc1), were you using the net-next branch with for these checksum tests?

I'm wondering if we have any checksum endianness bugs lurking in the upstream code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-880818349, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXS5JD5KCOXPAYQXR3TTX4BDNANCNFSM46FZEKNA .

[praupadhyaya]

Hi Mat,

Just tested with 5.14-rc2, and I could interwork successfully with my stack on MPTCPv1 -- checksums are fine too this time. So all looks good.

Regards -Prashant

On Fri, Jul 16, 2021 at 10:41 AM Prashant Upadhyaya @.***> wrote:

Hi Mat,

I was testing with 5.13-rc6. Will try the 5.14-rc1 as you indicated and will let you know the results. My MPTCPv1 stack is currently running on little-endian system (x86)

Regards -Prashant

On Thu, Jul 15, 2021 at 9:33 PM Mat Martineau @.***> wrote:

@praupadhyaya https://github.com/praupadhyaya, is your own MPTCPv1 stack on a big-endian or little-endian system? Also, checksum support was added after kernel v5.13 (now it is in v5.14-rc1), were you using the net-next branch with for these checksum tests?

I'm wondering if we have any checksum endianness bugs lurking in the upstream code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/multipath-tcp/mptcp/issues/427#issuecomment-880818349, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRHSXS5JD5KCOXPAYQXR3TTX4BDNANCNFSM46FZEKNA .

[matttbe]

Hi Prashant,

Thank you for your feedback.

So on your side, everything is OK with 5.14-rc2, right? (except IP_TRANSPARENT is not supported). But not with the out-of-tree kernel on top of 5.4, even without checksum, you have issues, right? We can then leave this ticket for the out-of-tree kernel.

Do not hesitate to add more details about your stack, share interop results and track interop regressions that can happen ;-)

[praupadhyaya]

Hi Mat,

Yes, your conclusions are correct. IP_TRANSPARENT is the only issue left (actually I am yet to test it with 5.14rc2 for IP_TRANSPARENT, so unless some conscious fix has gone in for this, this would still likely be an issue)

Regards -Prashant

On Tue, Jul 20, 2021 at 4:23 PM Matthieu Baerts @.***> wrote:

Hi Prashant,

Thank you for your feedback.

So on your side, everything is OK with 5.14-rc2, right? (except IP_TRANSPARENT is not supported). But not with the out-of-tree kernel on top of 5.4, even without checksum, you have issues, right? We can then leave this ticket for the out-of-tree kernel.

Do not hesitate to add more details about your stack, share interop results and track interop regressions that can happen ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[praupadhyaya]

Hi again Mat,

Just tested. So IP_TRANSPARENT is still a problem in 5.14-rc2.

Regards -Prashant

On Tue, Jul 20, 2021 at 4:38 PM Prashant Upadhyaya @.***> wrote:

Hi Mat,

Yes, your conclusions are correct. IP_TRANSPARENT is the only issue left (actually I am yet to test it with 5.14rc2 for IP_TRANSPARENT, so unless some conscious fix has gone in for this, this would still likely be an issue)

Regards -Prashant

On Tue, Jul 20, 2021 at 4:23 PM Matthieu Baerts @.***> wrote:

Hi Prashant,

Thank you for your feedback.

So on your side, everything is OK with 5.14-rc2, right? (except IP_TRANSPARENT is not supported). But not with the out-of-tree kernel on top of 5.4, even without checksum, you have issues, right? We can then leave this ticket for the out-of-tree kernel.

Do not hesitate to add more details about your stack, share interop results and track interop regressions that can happen ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[matttbe]

Hi Prashant,

Thank you for having looked. A new issue has been created on the MPTCP Upstream project: https://github.com/multipath-tcp/mptcp_net-next/issues/217

[cpaasch]

AFAICS, the issue on mptcp_write_mpcapable_data is still present.

On Matt's 3)'rd point: The code is indeed confusing, but it is correct AFAICS. Whenever mptcp_is_data_mpcapable is true, we treat the dss_offset to point directly to dlen + csum instead of the sequence-number.

[matttbe]

We can probably close this ticket thanks to @cpaasch 's patch: https://github.com/multipath-tcp/mptcp/commit/e0bed8c67bcaa21ac62bc83300ef0e4a3fa5c1c4