Read watermark

[Matthias247]

With the current design, a client is not able to indicate how many bytes it wants to read on a stream. It would simply do

let chunk = stream.pop().await?;

Now a certain concern here could occur if the peer sends a lot of small packets (which are "normal" in Quic with <= 1200 byte payloads):

• For every packet, the payload gets stored in the streams receive buffer
• The stream would notify the application code
• The application code would read it, but since it requires more data it gets blocked again

This could repeat a couple of times, and in each step we are switching between 2 tasks and are locking/unlocking a Mutex. E.g. to read 16kB - which is a reasonable size for a HTTP DATA frame - we might have 13 transitions between application task and QUIC task.

One way to optimize this would be to make sure we process as many packets for a connection as possible for unlocking the mutex and signaling the application task - which might be worthwhile to do anyway.

However we can also improve here on the API side: The client could be able to set a "Low Watermark" - the minimum amount of data that needs to be stored in the receive buffer in order for the wait handle to be awoken. For HTTP/3 DATA frames, the HTTP/3 library could set this one to the length of the remaining frame. This should minimize the amount of necessary wakeups.

If we implement this, I think the watermark needs to be a recommendation and not being strictly enforced. E.g. the QUIC library would need to be able to notify the reader if the receive buffer is (full / half full /etc) - even if the watermark hasn't been hit yet. API wise there are probably a couple of ways to do this.

We could add the API to just the watermark, and keep the pop APIs as is (and make them respect the watermark). E.g.

stream.set_receive_low_watermark(16*1024)?;
// Unblocks only if 16kB 
let chunk = stream.pop().await?;

or we could add an even new API, which just allows to "wait until watermark is hit" - and then make the pop API non-awaitable (similar to how the write API works):

stream.set_receive_low_watermark(16*1024)?;
// Unblocks only if 16kB 
stream.wait_ready().await?;
while !reached_16kb {
    let chunk = stream.pop()?; // No .await here
}

I think in order to determine whether this is useful we need to perform some benchmarking on how many of those context-switches we actually observe when reading HTTP/3. But I still determined it would be worthwhile to document it here.

Also @zz85 - since he's working on reading HTTP/3 frames.

[camshaft]

Implemented in #116, #129 and #160

[camshaft]

Sorry, it's actually not currently exposed so we need to figure out the right API for it.

Another option (and my preference) is to use the existing private request method:

stream
    .request()
    .with_low_watermark(123)
    .receive(&mut chunks)
    .await?;