Dealing with Congestion Control

[suhasHere]

There were few discussions that discussed on dealing with congestion and needing more thinking around decisions around how and when to drop things.

[kixelated]

I've been verballizing this quite often but here's my thoughts written down. There's a few problems with traditional TCP congestion control that is common in QUIC (Reno, CUBIC, BBR).

Bufferbloat Traditional loss-based congestion control (ex. CUBIC and Reno) is especially bad for live media. Bufferbloat is the tendency for routers, especially in emerging countries, to queue packets rather than drop them during congestion. These queued packets effectively cause head-of-line blocking (and thus latency), even if there's no packet loss. The sender loses the ability to decide which packets should be dropped or prioritized without some way to signal that decision on the wire at the IP layer.

Application-Limited The live media bitrate is gated by the encoder which most of the time will be lower than the maximum network bitrate (otherwise media is queued/dropped). This is especially a problem when delivering media frame-by-frame, as the sender is not able to fully saturate the congestion window outside of I-frames. Many congestion control algorithms (ex. CUBIC, Reno, BBR) can only increase the congestion window iff it's fully utilized, which is rare for live media and causes slow growth.

Consistent Throughput TCP congestion control algorithms are typically optimized for average throughput, not consistency. This is especially a problem with BBR, as it periodically switches to the PROBE_RTT state that effectively stops sending packets for an extended period. This gap causes starvation and in the live media case, depleting the jitter buffer which causes frame dropping or rebuffering.

[kixelated]

To address these issues, GCC is the standard congestion control in WebRTC-land. It requires transport wide CC. I don't have too much experience with the algorithm itself.

The idea is that the receiver reports the arrival timestamp for every packet. The sender can inspect the jitter between these packets to guess the pacing rate on intermediate routers. It's a similar concept to BBR, except it works at the packet level instead of the ACK level. This means it works when application-limited, unlike BBR.

QUIC ACKs includes the delay relative to the largest received sequence number. This means the sender can deduce the RTT for a single packet with each acknowledgement. This is more information than TCP, but I'm not sure if it's good enough.

In my opinion, there's a few improvements we should recommend to the QUIC WG. Specifying that the receiver ACK each packet immediately would be verbose but would be sufficient. Some way of batching the delay for each packet, much like how ACKs themselves are batched, would be an improvement.

[fluffy]

I think the closest thing that will work well for us is BBRV2 but with modifications. Of course the modifications would need to be done over in QUIC. Basically the problem with it right now is when it probes for RTT, it causes a temporary stop of media flow which kills the jitter buffers and makes for huge latency in the jitter buffers. I think the BBRv2 algorithm could be tweeked to still get this RTT without the pause.

[kixelated]

Yeah, I agree.

In fact, I think we can skip the BBR PROBE_RTT phase entirely because live video is sufficiently application-limited. It's meant to detect a local maxima caused by over-saturating the network, but that shouldn't be necessary since media will under-saturate the network.

[ianswett]

You definitely don't want to ACK every packet individually. If you want per-packet timestamps, I wrote a draft that was used by some internal projects. https://www.ietf.org/archive/id/draft-smith-quic-receive-ts-00.html

It can probably be improved, but it works with GCC and similar congestion controllers.

I agree that if you're app-limited often enough, you should be able to skip PROBE_RTT in BBRv2/3. I keep wanting to write a version of the BBR bandwidth estimator that uses timestamps, but it's hard for me to know how much better it'd be without just doing it.

This is an interesting topic, but is there a clear ack for the QUIC WG or text we should add to the existing draft? I'm inclined to park this for now and wait for more deployment experience.

[kixelated]

This is one area where WebRTC has advantage over MoQ. While I don't think the draft should have any text, it would be good to have a plan we can point to. Getting your timestamp draft adopted eventually sounds like a good plan to me.

[PercyLau]

You definitely don't want to ACK every packet individually. If you want per-packet timestamps, I wrote a draft that was used by some internal projects. https://www.ietf.org/archive/id/draft-smith-quic-receive-ts-00.html

It can probably be improved, but it works with GCC and similar congestion controllers.

I agree that if you're app-limited often enough, you should be able to skip PROBE_RTT in BBRv2/3. I keep wanting to write a version of the BBR bandwidth estimator that uses timestamps, but it's hard for me to know how much better it'd be without just doing it.

This is an interesting topic, but is there a clear ack for the QUIC WG or text we should add to the existing draft? I'm inclined to park this for now and wait for more deployment experience.

It is possible to construct a case that ack every packet is not necessary, but vice versa. Acknowledgement mechanism seems to be some kind of compression of network state information. E.g., current ack format can be interpreted as a way using delta encoding to compresse a simplified black-box model which only determines if a packet is lost or not. Further, the upper application layer may have an impact on what kind of compression algorithm we choose to represent. We don't know the accurate state changes of upper application layer under different packet receiving patterns. In other words, we lack a detailed analytic framework from an information-theory perspective.

[alvestrand]

Determining a proper information-theoretic approach for building new CC mechanisms is probably the business of CCWG, not the business of MOQ.

(The experience with building Goog-CC / Transport-wide is that knowing the delay of every packet is very helpful in determining what's happening in the network - but the information can be easily packed up in bundles, so it's not necessary to have an ACK for every packet.)

[afrind]

Ian will write a PR with best practice guidance for interaction with congestion control.