All ACKs carry TCP ECN-Echo (ECE) in a PRAGUE stream

[MatteoGuarna]

Hi all,

I was doing some experiments in lab with L4S scenario and came across a strange behavior.

In few words, I noticed that in a TCP Prague stream, all generated ACKs (in any scenario, even in a low-load network with no congestion)

have the TCP ECN-Echo (ECE) flag set to 1, and not only in response to packets marked with the IP ECN flag "CE" as expected.

Here, a little bit of context.

I'm trying to test the L4S suite by sending TCP data between two linux hosts (act as client and server) using Prague/Cubic, with a forwarding AQM linux router equipped with the dualpi2 feature in between. My objective is to compare L4S/Prague against Cubic on a congested link.

Both hosts and the router utilize the modified kernel from this repository, and both the Prague and dualpi2 implementations I relied on are sourced from here.

We generate two data flows (with iPerf3) at 100 Mb/s from Server to Client, one flow is with Prague and the other is with Cubic Congestion control.

While performing the tests, a reasonable number of PRAGUE packets are marked with CE information by the router to warn the server of an incoming congestion, but unexpectedly all PRAGUE ACKs carry the TCP ECN-Echo (ECE) flag. I would expect to see this flag only on the ACKs in response to the CE marked packets.

This results into a wrong signaling of network congestion, which prevents PRAGUE from competing fairly with CUBIC: in fact the CUBIC average throughput occupies about 80% of the channel’s capacity, leaving to PRAGUE less than 20% of bandwidth.

The phenomeon can also be seen while generating traffic with low-load network and no congestions happening.

I might have, of course, made some mistakes in the system's configuration. I followed the instructions in repository's README.

All Prague-generated traffic is correctly marked as L4S capable (with ECT(1)), indicating that my setup should be correct.

Thank you in advance for your help, and apologize if I have overlooked something.

[minuscat]

Hi Matteo, It looks that your configuration are correct (based on my reading) but your setup make TCP Prague and DP2 not worked expected. I would suggest to start with (1) only Prague traffic flow(s) with DP2 under the rate limitation to see exactly your packet marking by DP2 work or not and whether TCP Prague can well saturate the limited rate, (2) check qdisc statistics on DP2 to see how many packets are marked by DP2, and (3) make sure you do not have another bottleneck doing 100% packet marking. And one more thing, remember to configure using FQ in qdisc at both sender and receiver side to make sure pacing work and disable all TSO, GSO, GRO for the related interface. using ethtool. If issue persist, I would suggest you to provide your network setup or other logs/trace fore debug the setup.

[MatteoGuarna]

Please forgive me for the delay in the answer, I am only now able to reply with my observations and findings:

First of all I wanted to thank you for your help; in fact I disabled the offloading capabilities on the endpoints (the only thing I had not tried yet) and finally I see (at least at a "macroscopic" level) the behaviour I was expecting.

A little plant of both my configuration and findings are attached to this message, which may be useful to follow through with my question. testplant_l4s_issue_github.pdf

Flow measurements (the good news):

As a matter of fact, upon several tests I can see that with two parallel flows congesting on the same router (PRAGUE + CUBIC) the delay with the PRAGUE flow is smaller and more stable (smaller jitter) than CUBIC, which is the main point I expected to achieve.

Basically, by generating two flows on 100Mbps links each (one for CUBIC and another for PRAGUE) through a 100Mbps bottleneck on a router with DP2 (on a network with a 30ms fixed delay) I get the following results:

• PRAGUE traffic has on average 63.4 Mbps throughput

• CUBIC traffic has on average 28.6 Mbps throughput

• PRAGUE half-trip delay is constant and averaging 30.7 ms

• CUBIC half-trip delay swings mostly between 33 and 35 ms

While the delay and the jitters are exactly as I expected the flows to work, the more-than-double rate of Prague is a little perplexing to me, for I expected the DP2 to act more fairly. This is in fact of great interest for me, it's the main point of my studies so I would like to ask you if you have any possible explanations for this behaviour.

Packet Inspection (the bad news):

In order to take a deeper look at the functioning of the protocol I performed packet inspection by deploying Wireshark on the router's interfaces.

Just as a premise, I checked both flows individually and scanned their headers' fields. CUBIC behaves exactly as expected, while, as I stated in my previous message, the ECN Echo flag (ECE) is still present on all PRAGUE packets and ACKs. All packets coming from the sender also carry the CWR flag. This is odd to me.

Still, with both flows running simultaneously, the following happens:

• The router still marks a (small but in my opinion correct) fraction of 8% of the PRAGUE packets downstream of the congestion with CE which is good and reassures me that the router works as it should.

• All PRAGUE packets are sent and reach the receiver with the ECE flag on, but the ACKs are not all ECE marked anymore, as it happened before I turned off the offloading capabilities:

• I see in fact that while around 55% of the ACKs still carry the ECE flag, now the remaining 45% are ECE=0. THIS IS THE ONLY THING THAT CHANGED in response to the disabilitation of the offloading, but it seems that it's enough to make my testplant work in some way.

• I also still see that all packets generated by the sender carry the CWR flag, as they did before, while I would have expected that to happen only on a fraction of the packets. The flag is though CWR=1 also if the flow is alone and no congestion occurs.

Summary:

In conclusion, I reported these three issues in a Prague flow:

• In a bottleneck scenario with a RENO-compatible flow, the PRAGUE flow doesn't share the bandwith so fairly as it should be.
• The default setting for the ECE flag seems to be ECE=1, and in a bottleneck scenario, in the face of a "normal" rate of CE packets (8%), a huge percentage (45%) switch their flag (although from 1 to 0, and not the reverse).
• All PRAGUE packets carry the TCP CWR flag, also when no congestion occurs.

I believe I might be doing some mistakes somewhere, and I thought it better to provide you with a more detailed description of my issues, and I hope we could sort this out together.

I thank you a lot in advance for your kindness and your willingness

Matteo

[rjmcmahon]

A bit off topic, but iperf 2 has lots of latency features include one way delay (OWD) metrics. It also supports passing the client CCA to both ends and supports dual CCAs for a simple Classic/L4S test. Iperf 3 is mostly a capacity tool and less a latency tool.

[moeller0]

It is a well known and documented fact, that: TCP Prague has a stronger RTT-bias than cubic: https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667 TCP Prague and Cubic do not share equally in a FIFO: https://camo.githubusercontent.com/572896b1f713045d3e146dd32d3431a441d1b6a442e0f3a101d39660094bf93d/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667 At lowish RTTs DualQ gives a larger capacity share to the L-queue compared to the C-queue (team L4S declared since the C-queue is technically not starved this to be inside their design parameters, IIRC). https://camo.githubusercontent.com/a83b3e8c3aa1824e4ff38b5d0d9136079f431c2e917bb347c15b8b17d64b6a32/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667 See https://github.com/heistp/l4s-tests for details.

The shortcomings of the AQM can likely be ameliorated by switching to a flow queueing scheduler instead of the simple dual-queue design. But the increased RTT-bias over Cubic (and I believe Reno) seem to be properties of TCP Prague that have not been fixed.... (but see the modifications in TCP Prague to modify the RTT dependency; in my book the fact that these are configurable and do not actually remove the RTT-bias indicate to me that RTT-bias is not a fixed issue in TCP Prague, but opinions on that differ).

[MatteoGuarna]

A bit off topic, but iperf 2 has lots of latency features include one way delay (OWD) metrics. It also supports passing the client CCA to both ends and supports dual CCAs for a simple Classic/L4S test. Iperf 3 is mostly a capacity tool and less a latency tool.

I was actually trying to check the fairness in the sharing of the channel, but I wasn't aware of iperf2's latency features. Thanks a lot for your advice, I checked it out, it might turn out very useful!

[MatteoGuarna]

It is a well known and documented fact, that: TCP Prague has a stronger RTT-bias than cubic: https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667 TCP Prague and Cubic do not share equally in a FIFO: https://camo.githubusercontent.com/572896b1f713045d3e146dd32d3431a441d1b6a442e0f3a101d39660094bf93d/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667 At lowish RTTs DualQ gives a larger capacity share to the L-queue compared to the C-queue (team L4S declared since the C-queue is technically not starved this to be inside their design parameters, IIRC). https://camo.githubusercontent.com/a83b3e8c3aa1824e4ff38b5d0d9136079f431c2e917bb347c15b8b17d64b6a32/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667 See https://github.com/heistp/l4s-tests for details.

The shortcomings of the AQM can likely be ameliorated by switching to a flow queueing scheduler instead of the simple dual-queue design. But the increased RTT-bias over Cubic (and I believe Reno) seem to be properties of TCP Prague that have not been fixed.... (but see the modifications in TCP Prague to modify the RTT dependency; in my book the fact that these are configurable and do not actually remove the RTT-bias indicate to me that RTT-bias is not a fixed issue in TCP Prague, but opinions on that differ).

Really helpful, this was of great comfort to hear! Thanks a lot for your help!