BBR low sending rate when application-limited

[alexrabi]

Seems like there is a problem in the BBR implementation if the sender is application-limited rather than network limited. If the sender only sporadically has data to send, it appears that the CWND becomes very small. This severely limits the sending rate once the application actually has data to send.

A similar problem has already been described in https://github.com/private-octopus/picoquic/issues/1499#issuecomment-1645750380

[huitema]

This is a classic congestion control issue. BBR sets the pacing rate based on the highest value observed in the recent past. The definition of recent is debatable: if the highest rate was observed a long time ago, then the network conditions have probably changed and the observation is not valid anymore. If rates observed too recently are discarded, we get the situation that you describe.

Can you characterize the sending pattern of your applications? That would help setting a simulation, and understanding the issue more precisely.

[alexrabi]

BBR should ignore rate samples if it is application-limited, looking at the ietf draft, though I'm not sure if the picoquic implementation does this correctly.

The sending pattern is literally TCP traffic; the problem is observed when tunneling a TCP connection over QUIC datagrams. More specifically, the problem occurs when TCP is sending pure TCP ACKs, so this would be able to be simulated by sending one or more small datagrams roughly every round-trip-time (or slightly longer).

[alexrabi]

After doing some more tests, it looks this problem only happens when using multipath, and there is too little data to send to use more than one path. If there is only a single path available, there is no issue (whether multipath was negotiated or not).

[huitema]

Thanks for the precision. We test code includes test cases of limited senders, which are passing, but they obviously do not include your multipath configuration.

There is indeed a special case if there is no traffic at all on a path for some time. We probably need to add a test case for that.

[huitema]

Trying to understand your scenario. Apparently:

• you are testing a VPN connection, with traffic carried as datagrams over multipath QUIC.
• for an extended period, the traffic from the server corresponds to TCP data packets, and the traffic from the client consists of TCP ACK packets.
• somehow, the TCP ACK packets are sent on just one of the available paths -- maybe because one path is preferred by the scheduler.
• after some time, the client side of the connection wakes up and start sending TCP data packets.
• the expectation is that those packets will be spread over both paths, but at this point BBR parameters are such that very little traffic is sent on the second path.

Is that a correct assessment?

Any idea of the duration of the "low traffic" interval?

[alexrabi]

• The actual use case is ATSSS/hybrid access, though for all intents and purposes it will act like a VPN.
• Correct.
• Yes. This issue seems to occur whether or not datagram path affinity is set, though it seems to be more likely if the datagrams are mapped to a specific path (i.e. the secondary path is not used at all).
• Yes, but not necessarily. The issue remains even when the traffic from the client consists of pure ACKs throughout. The cwnd and pacing rate of the primary path seems to become so low that the TCP sender essentially starves due to the TCP ACKs being heavily delayed.
• It is actually the BBR parameters on the primary path that seems to be the issue; the secondary path seems fine.

The duration of the "low traffic" interval could essentially be infinite (e.g. a TCP flow generated by a traffic generator such as iperf), or at the very least very long (e.g. a large file download).

Funnily enough, it appears that if one creates a completely separate flow of datagrams that is only scheduled on the secondary path, the parameters on the primary path are affected (positively).

I have tried other congestion control algorithms (reno, cubic), and they do not display the same issue, though those algorithms are less than ideal for this use case for other reasons.

[huitema]

OK, I think I understand your scenario. It seems that there is a weird feedback loop happening, and I have to dig into it.

Just one more question. You say that the client only sends ACK. Shall I assume that this is true for the entire connection, i.e., the client is never sending more than a few ACK per RTT?

[alexrabi]

I think that that assumption would be fine. The way I am running things right now there is a tiny bit of negotiation between the client and server at the start and end of the connection (i.e. I am running iperf), though the client is limited to sending ACKs throughout the rest of the connection.

[huitema]

@alexrabi PR #1548 ought to solve you issue. I could not reproduce your exact scenario, but the PR fixes the handling of application limited scenarios in BBR, and also the accounting of small packets in the pacing code. I did add a test in which the traffic consists solely of small packets, and the PR causes a very significant improvement in the processing of these small packets.

[alexrabi]

@huitema After running a number of tests in my setup I can confirm that PR #1548 fixes the issue.

[huitema]

Thanks for the tests!