SUBSCRIBE with absolute ranges and flow control

[vasilvv]

This is another SUBSCRIBE-vs-FETCH issue, but with a specific implementation aspect of it.

Assume I have a group-per-stream track. I can subscribe to groups 1000 to 2000; it is entirely plausible that the publisher has all of those in cache immediately available, so in a naive implementation it can just open 1000 streams and start sending things; it would run into a flow control, and starve everything else on the connection.

We could set some limit on how many streams an individual subscription has, and that would be one way of solving this: if you set deliver order to ASC, you would process old streams first, and once those are processed (something we don't actually have a signal either), you'd start sending the next group. Is this what we actually want?

[afrind]

Individual Comment:

+1 to some kind of subscription level flow control or rate control when behind live head. It's a problem for absolute start also. The moxygen date server has groups going back to 1970, and I've hit this problem when I specify the wrong start group (eg: 0)

[wilaw]

Several solutions jump to mind:

1. Setting max-concurrent-streams is an interesting approach and would be simpler to implement for the relay. It might still lead to starvation of competing traffic if not set low enough, however as long as it's applied per connection then the subscriber would be DOSing themself and would have the means to change it.

2. It might be useful for players fetching cached (or instantly available) content to be able to specify a maximum per-connection send rate. This would allows the player to request 1000-2000 but limit the throughput to say 105% of the encoded bitrate, which would give them a slowly incrementing forward buffer. Would it be computationally expensive for the relay to implement this throttling , as it might need to to micro-bursts that on aggregate provide the target throughput? We wouldn't want to reduce the egress capacity of a relay by forcing it to implement throttling.

3. A third approach is that we don't do anything. A naive player would subscribe to range 1000-2000 and flood the connection. However a smarter player could request 1000-1010, 1011-1020 .. etc and thereby manage the peak bursts. Relays could apply their own protection limits for concurrent streams when sending data.

[fluffy]

So when doing a fetch ( and by fetch I mean any sort of request for old or already existing objects), we have a huge issue in how to deal with flow control and more importantly how the priority of that compete with priority of future data. If I have one stream of live edge data, and 10,000 streams of old groups, I don't think that would meet any use case I care about.

Seems like a large topic worthy of in person discussion

[TimEvens]

My two-cents:

• When servicing requested objects from the past/cache; the transmission should use per-track to deliver those objects.
• If there is an end-range requested, then end of track/stream is end of retrieval.
• If there is no end-range and the client is expecting to tap into the live stream at the same time as receiving cached objects, then the live stream will follow the publisher delivery mode of using stream per-track, per-group, per-object, or datagram.
• It's a bit slippery when it comes to datagram and the mix between cache and live. I still say use per-track to deliver cache, KISS and use delivery mode of the live stream...
• When delivering the live feed, it won't have the cache of objects that would cause a flood of streams.

The above being said, there is a challenge with flow control of using MOQT with relay-to-relay peering sessions. Regardless of cache, we'll have this challenge if every producer uses per-object with live data flows. The aggregated QUIC connection used between two relays could easily have in the millions of active streams. Is that ideal with QUIC?

[vasilvv]

One observation I have is that even if we do flow control, there is a bit of awkward dynamic here with doing fetches of large ranges.

Let's say we have 10 clients, and each of them requests same 1000 groups simultaneously. Ideally, those would all arrive at the same time, and we would forward them immediately. Now, assume one client can consume 100 groups per second, the second client can consume 50 groups per second, the third can do 25, and so on. We probably don't want a system in which relay has to hold all 1000 objects while the slowest client receives them. We can deal with this in at least four ways:

1. Throttle upstream subscription to slowest. This is extremely suboptimal since it means that the slowest client will affect all others. This also doesn't work when the requests only partially overlap.
2. Require the clients only request data in small chunks. If the client requests something and unable to receive that in time, subscribe would fail and client would have to resubscribe at the updated range.
3. Make the relay request data in small chunks upon cache miss. This would mean that a single subscribe would potentially result in hundreds of upstream subscribes spread over time.
4. Do not deduplicate fetches at all. This is bad, since we'd be re-requesting data that we have in local cache. It would also be a downgrade from HLS/DASH.

[afrind]

Individual Comment:

Isn't the scenario you describe similar to HTTP cache fill?

You probably shouldn't throttle cache fill to the slowest client, and you shouldn't OOM your relay buffering for them either. We do something like:

Fill at line rate and write back to an external cache Hold a fixed size in memory buffer per client If the client exceeds the buffer, switch them to filling from the external cache (or go back upstream) when they drain it.

[vasilvv]

Isn't the scenario you describe similar to HTTP cache fill?

Kind of, except you're cache-filling a bunch of potentially overlapping range requests for a single resource.

[fluffy]

On the 4 things we could do above. I agree with victor than 1 and 4 sound really bad.