define a new stream multiplexer

[marten-seemann]

yamux is bad, and mplex is even worse. We need a better stream muxer for TCP connections.

Specifically, we need a muxer that has:

• back pressure on the stream level, i.e. a limit how much outstanding data can be sent on a single stream
• back pressure on the connection level, i.e. a limit how much outstanding data can be sent on the entire connection (typically, this limit will be significantly lower than the number of streams multiplied by the stream data limit)
• back pressure for new streams, i.e. a limit how many streams can be active at any given point

Ideally, all these limits can be set dynamically during the lifetime of a connection (e.g. allowing with small limits, and increasing them later).

The QUIC stream multiplexer fulfills all these properties, and we know that it works well. We could adapt it to work over TCP.

I've started writing up how this could look like: https://hackmd.io/JXgEHrA5SGy7ioKCp-OnGQ. This document will need a lot more love before we can implement something. We will also need to coordinate the implementation work across languages: This effort only makes sense if we have implementations in Go, Rust and JavaScript. Once we have those, we can start phasing out old stream multiplexers, for example by reducing their limits (so performance decreases, but we don't break connectivity to legacy nodes).

[Stebalien]

Note: Yamux has backpressure for new streams (unacked), it just doesn't have a total "stream" limit. But yeah, the latter is likely more flexible than the former.

Other desirable (but not critical) properties:

• Efficient. IIRC, yamux headers aren't quite as compact as they could be.
• One-way. QUIC supports uni-directional streams, yamux doesn't.

Possible paths:

• Add the necessary backpressure to yamux as-is. Honestly, this isn't a huge change and is probably the fastest way forward.
• Use the spdy stream multiplexer. We tried this in the past but dropped it because we thought it had some bugs (but those bugs were unrelated).

[mxinden]

Efficient. IIRC, yamux headers aren't quite as compact as they could be.

Yamux headers are 12 bytes. Say a frame is on average 64 KiB of goodput (totally made up), the overhead would be 12 ÷ (64×1024) = 0.000183105. I would argue that this overhead is negligible.

Add the necessary backpressure to yamux as-is. Honestly, this isn't a huge change and is probably the fastest way forward.

:+1:

[Stebalien]

Yamux headers are 12 bytes. Say a frame is on average 64 KiB of goodput (totally made up), the overhead would be 12 ÷ (64×1024) = 0.000183105. I would argue that this overhead is negligible.

Well.... we have some pretty small writes (20-60 bytes). We have some (disabled) logic for write coalescing, but that merged writes from multiple streams.

But yeah, it probably doesn't matter too much given that, if we're bandwidth constrained, the majority of the bandwidth will be taken up by large writes.

[marten-seemann]

Note: Yamux has backpressure for new streams (unacked), it just doesn't have a total "stream" limit.

I'd phrase it this way: Yamux has the concept of unacked streams, but no way to reasonably act on it (I don't consider the stream muxer resetting a stream to be reasonable).

Say a frame is on average 64 KiB of goodput (totally made up), the overhead would be 12 ÷ (64×1024) = 0.000183105

That frame size is way too large. If you're committing to sending 64 KB without the option to send any other frame (be it a control frame or data for another stream), you're going to have a very unresponsive stream muxer.

We can bring down the frame size of a STREAM frame to 2 bytes in the best case (1 type byte + varint stream ID).

[Stebalien]

I'd phrase it this way: Yamux has the concept of unacked streams, but no way to reasonably act on it (I don't consider the stream muxer resetting a stream to be reasonable).

Well, as long as both sides agree on the same number of unacked streams, you can implement backpressure that way. But yeah, a negotiated value would be better.

If you're committing to sending 64 KB without the option to send any other frame (be it a control frame or data for another stream), you're going to have a very unresponsive stream muxer.

Really? That's the current "max message size". We can make it smaller, but I'm worried that we'll lose performance when, e.g., jumbo-packets are available. Although I guess that's rather unusual.