Implement subscribe intent

[huitema]

QUICR aims to unify real time transmission (e.g., Webex call), life streaming (e.g., watching a football game) and non-real-time streaming (e.g., watching a video recording). These different scenarios have different caching requirements:

• for stored media, playback should start from the beginning of the capture;
• for real time or life streaming media, playback should start at the current time.

We also need to anticipate cases when the client wants to start playback at a specific time, for example after doing a fast forward on stored media, or after losing and re-establishing a connection.

[huitema]

@suhasHere -- here is a description of the required changes. Implementation requires changing the format of the "Request" message, its processing, and testing the scenario.

For the "request" message, we need to add an "intent" attribute that can encode whether to start at the current time, as well as the start group and start object. Probably need to encode that as an "intent" attribute, with three properties. That means:

• define the syntax, rev the version number and the ALPN, and update the description of the "Request" message in "protocol.md"
• implement the changes in the reserve/encode and decode functions for the "request" message in proto.c
• update the test of protocol messages in "proto_test.c" to test the new format.
• update the quicrq_subscribe_object_stream API to take the new parameters and encode them in the request message.

The processing implies:

• update the processing of incoming actions QUICRQ_ACTION_REQUEST_STREAM and QUICRQ_ACTION_REQUEST_DATAGRAM in quicrq.c
• update the processing of quicrq_subscribe_local_media to handle the new parameters. May need to find a way to pass then to the media handler through the function pointer subscribe_fn
• update the concrete implementations of subscribe_fn in relay.c (quicrq_relay_publisher_subscribe) and object_source.c (quicrq_media_object_publisher_subscribe). Currently, these functions set the start point in the media stream context (stream_ctx) to the start value in the local cache context. Instead, they should put the values to the adequate start point, as derived from object intent.

The validation implies:

• create a set of test cases, probably derived from the "triangle" tests, in which we test the various "subscribe intent" scenarios, and verify that the requested media is properly received, starting at the expected start point.

[TimEvens]

@huitema ,

Coming in a bit late and I do not have all the context...

for stored media, playback should start from the beginning of the capture; for real time or life streaming media, playback should start at the current time.

Beginning of the capture implies state tracking and replicated storage of published content. This introduces complexities, such as:

• multiple relays, mesh of relays with optimized paths; not all relays will have the same cache. Startup of a new relay connection introduces cache sync challenges...
• memory usage... Ideally, we should keep memory to 4GB or less per relay instance.
• If each stream has a start and end, which is very similar to a file, it would be pretty easy to exploit and cause a denial of service in terms of memory, cpu, ...
• Playback often needs to be paced, otherwise it could introduce micro-busrting on subscription with a sudden dump of X messages. Handling this internally within the relay adds additional state tracking per subscriber...
• Per subscriber offset tracking on playback due to subscribers running at different speeds
• Massive duplication cause by subscriber reconnects/flapping/... Duplication can be handled by offset tracking per subscriber, but that complicates things more with subscriber management
• Subscriber may not be able to catch up to the real-time stream of data. For example, 10,000,000 messages to send to new subscriber, but subscriber is so slow that real-time never is realized.
• Subscriptions have wildcard(s) and can be include more than one stream (e.g., file). The amount of data that needs to be stored and transmitted to a subscriber (client) is likely too vast and not needed. How we deliver last X messages from cache will need to be based on matched (due to wildcard(s)) names. Maybe a simple division of X messages by matched names.

For media, we should use the latter, which I'm proposing is the default behavior. A new subscriber only receives new messages, none from the cache. Subscription can have knobs to change this behavior, but rather than go down the rat hole of subscription management with per-subscriber/offsets (e.g., Kafka consumer management), we can instead provide a simple tail approach. The subscriber can subscribe with a uint16 value to indicate how many messages it would like to consume from the cache on new subscription. There would be no guarantees on the cache having the requested number of last messages. Instead it's more of "I'll accept this much". The messages would always be delivered ascending based on receiver order. During cache transmission, real-time messages are enqueued and delivered after X messages have been transmitted.

I'm on the fence with allowing the subscriber to change the order of cache message delivery. To keep things simple and to meet 99% of the use-cases, we don't need a knob for this. Messages should always be delivered in first-in order of the cache starting back X number of messages.

The cache is extremely important to de-dup messages from other relays. I'm advocating that the relays themselves store content for a limited time to support primarily de-duplication. Tailing the past messages is a small feature benefit from the relays. Instead, if the client really wants to stream past content, even from the beginning, then we should have a side application that facilitates that. A good example of this is with iMessage. Past content is not made available to other devices unless you enable cloud storage. We could have a mini-recording service that records published messages, likely at the source, and then stores that on disk for replay. Out-of-band signaling could trigger a replay. Out-of-band signaling could be still via the relays, such as to a well known name to request a replay to a given subscriber. We certainly would not want one subscriber to cause a replay to all subscribers matching the name (wildcard or not).

[huitema]

@TimEvens we have conflicting specifications. On one hand, yes, for real time applications the playback should start with new data. On the other hand, QUICR aims at unifying real time and streaming, i.e., subsume both the Webex and the DASH use cases. Maybe we need to add a property to the media source, checking whether the source is real-time or streaming. Assuming we can do that, then we have different caching requirements.

For real-time communication, the purpose of the cache is indeed to de-dup, and also to facilitate a clean start when the client connects, or re-connects after a break. The cache will also serve to accumulate data that cannot be immediately forwarded, e.g., stream data that are received as out of order datagram but maybe relayed as in sequence streams. Also, if multiple clients read the same media, the cache will serve them data as they can process it, maybe with different congestion controls applied to each client.

The specification I got was that data delivered to a client shall start at a "group of block" boundary. This ensures that the client will be able to render the data, and that the video will not look like a mosaic of squares. That implies that relays need to cache at least the current group of block. They may need to cache more data for clients that are still processing the previous blocks -- each client may be at a different reading point in the stream.

The relays do not currently cache "all the stream until the end". The cached data is dropped based on time to live. Yes, we may want to purge the cache more aggressively, e.g., drop everything before the most recent group of blocks and before the reading point of currently connected clients. That's a bit of a different issue than subscribe intent.

[suhasHere]

Since the focus is on live stream and interactive media, the cache lifetime is in range of 30s to 2 minute (hand waving since it depends on other dynamics of the system , cache vendor and so on). Anything that's outside of the live range is discarded and a recording server (which is also another subscriber) is responsible for dealing with such use-cases and not MoQ

Subscribe intent solves few issues like

• a new client joins in mid-session, he doesn't need to know anything too far behind to be effective in the session
• a client looses his connection temporarily and comes back online and wants to catch up to the on-going stream
• and couple of similar timing senstive use-cases

[TimEvens]

@huitema ,

"The specification I got was that data delivered to a client shall start at a "group of block" boundary. This ensures that the client will be able to render the data, and that the video will not look like a mosaic of squares. That implies that relays need to cache at least the current group of block. They may need to cache more data for clients that are still processing the previous blocks -- each client may be at a different reading point in the stream."

I need to talk with some of the media codec folks, but I suspect the relay keeping track of "group of block" data, is no longer required. Frames/packets/messages that represent only parts of the image can be dropped on replay/transmission. Nobody wants to see a partial image (e.g., various rendered blocks without complete image).

Having the relay keep track of "start of frame" (full image render) (aka group of block) is expensive and likely not required. It's also very specific in being able to identify start of block... Re-transmission is different than start of flow. Start of flow can ignore/drop messages until full render comes across. If it doesn't come within allotted time, the app can handle refresh. It does this today, regardless of the relay.

Re-transmission is really where things can be improved to handle lossy networks. Recovery is where it takes a while and directly impacts customer perceived quality.

[TimEvens]

@suhasHere, I'm not sure on the use of intent as intent often involves understanding/context/etc. IMHO, intent is related to the intent of [human, automation, ...], which is often context based. Subscriber intent would normally be involved with some level of analytics, did they want the past 10 minutes, 15 minutes, hour, etc. Based on analytics, we can see that Tim likes to resume to the past 15 minutes and therefore has the intent to stream the last 15 minutes and catch-up to real-time. Mo mentioned another use-case, where he would like to non-interactively replay the past hour with no intention of catching up to live stream.

Intent detection, context awareness, deciphering, etc. IMO, does not scale for a high performing relay network. Replaying data from the past hour falls under a recording service, to which could unicast or relay (pub/sub) stream a recording, including recording followed by live. Intent to instruction (api call as an example) can be performed out-of-band. I believe that's how Siri, alexa and others function. Take what the person says and translate that into prescribed instruction sets based on user intent/context/analytics/etc. At the end of the day, the backend system is not intent based, it's very much driven by specific API/RPC/instructions.

Intent is often abstract, in what does the API call really want, or what does this human want? Start of frame is binary and doesn't need to involve intent. Intent would be something like "I would like to go back 30 minutes and catch-up to live-stream, using spead times X to get to live streaming so I can interact with the live call." The intent requires a series of backend changes to facilitate it. It also requires the backend to understand the intent request, which is often in a vague request form that needs to be mapped to instructions. RPC/API calls to replay at X, start of group or not, is not really intent IMHO. It's a very specific instruction.

[huitema]

Tim, you are really saying that we should not try to unify streaming and real-time too much. We would have real-time media streams. If a user wants a replay, they will get it from a streaming server that has recorded the call. That streaming server would server something like DASH, i.e., series of small segments that can be played independently.

That would simplify the relation between "caching" and "intent". If the client connects to a real time video, it gets whatever the "most recent" means. If the client wants a replay, it connects to the replay server -- different media name, different caching strategy, etc.

[suhasHere]

@TimEvens Subscribe Intent doesn't fall into recording but a quick few seconds back in time catchup as mentioned earlier. Like Beginning of the current group or anything in the cache. CDNs caching requirements for realtime or live streaming media will come into play on how deep it can be. In general we are talking about <1s at the minimum to 5s at the maximum

[huitema]

I think that @TimEvens is right. It makes a lot of sense to assume that, by default, "GET real-time-media-X" provides the current-time media, not the cached media of minutes ago. The exception would be streaming, in which "GET movie-X-part-5" always provides exactly that part. The choice between one behavior and the other is not made by the client, but is a function of the media type.

For the prototype, I am not sure that we want to demonstrate playing movies or other recording. Caching realtime media is simple: cache the GOB that are not read by all currently connected clients; when a new client connects, always return the current GOB (and accelerate until we synch or reach the next one). Caching streaming media is more like caching web pages: cache a whole segment or cache nothing; absolutely no point in caching the last part of a segment if the previous part is missing.

Thus, for simplicity, I propose splitting this issue in two:

1) Change the current behavior to "real time", and assume that "intent" is always "latest GOB, latest object", 2) Add explicit support for "streaming segments", perhaps as a second PR.

The good news with that is that changing the code is simple. We have to:

1) Update cache management to be dependent on the real-time or not nature of the media stream. 2) If real-time, only cache the currently active GOB. This will replace the "purge old objects" code. 3) If real time, When connecting a client to a cache, serve the first object in the current GOB, then skip over all objects that "can be dropped" until reaching either the most recent object or the next GOB. 4) If media is streaming instead of real time, keep caching whole stream until current clients have been served to the end, and adopt drop-all-or-nothing cache management after that. 5) If media is streaming instead of real time, connect new clients to the exact beginning of the stream.

This is pretty straightforward to implement -- just need to add a specification of "real time or streaming" when creating a media source. I will also need to modify the tests. Most of them are written for the streaming behavior, except those used to test partial caching. Those ones are a great fit for testing the "real time" behavior.

[suhasHere]

Subscribe intent has 3 modes defined in the spec today as below. These were added to be possible to streaming/interactive and messaging kind of flows

immediate: Deliver any new objects it receives that match the name.

catch_up: Deliver any new objects it receives and in addition send any previous objects it has received that matches the name.

wait_up: Wait until an object that has a objectId that matches the name is received then start sending any objects that match the name.

Yes, we can adapt these to more clearer and update it as appropriate. We should separate the relay caching strategies from the subscribe messages and the corresponding hints.

Also, the data being published specifies a TTL that defines the liveness nature of the data and that's what caches can use to determine the cache duration. Ofcourse, cache local policies can always overwrite such a TTL.

I feel marking something as real-time or live-stream or vod stream or something similar leads to brittleness at the application layer from source to the target and also leads to assumptions in the protocol.

[huitema]

I think that there is no big difference between the 3 policies that you propose. Look at the proposed simplification: send the first object of the most recent GOB, then skip all that can be skipped until synch. Compare to:

• immediate: in traditional video, the only difference is the first object of GOB, typically the I-Frame. Without that frame, the "most recent object" cannot only be replayed as a bad mosaic of squares. For audio, all frames are marked as beginning of GOB, so last GOB is the last frame.
• catch up: in traditional video, the proposed policy will just catch up.
• wait up: needs to be expressed as "start at this GOB and that Object number". But how is the client going to know that? That only happens if the client has already received up to GOB X, Object N, i.e., in case of a reconnect. But if the current cache starts at GOB Y, Y > N, then the relay will start sending from there.

So I think we can certainly merge immediate and catch-up into a single default behavior, and I am not sure of the practical use of wait up.

[suhasHere]

I am not against the simplification. I just think a cache strategy is making application decision. Cache returns everything from beginning of the group of object and it is left to application to use it or drop it or drop few of it. Here I am considering the flags to guide for the application to make such a decision.

This will make the cache logic not tied to video (IDR, P,P, B, P, P, B, ...) or audio (1 segment vs ML audio multiple sample vs chat ) or something different that we haven't thought about

[huitema]

Another point is the expected response of the client. Suppose that the client is looking to "GET movie-X-part-5". The only useful response there is for the relay to begin streaming from the beginning of this segment. If the relay simply started from some point in the middle, the user would miss a part of the movie that they are watching. That also means the relay has only two choices for this media: cache the whole segment, or cache nothing.

That means the relay has to adapt its caching strategy to the "type of media": for some media, it has to cache everything or cache nothing; for others, it has to implement "real time caching", i.e., only keep the most recent blocks. The relay cannot learn that from the client request, because the cache is populated before that request. Thus, the relay has to learn that as a property of the media source. In our prototype, that will be a parameter in the "create source" API. That parameter will have to be passed to the relays in the "POST" command for user provided data, or in the response to the GET command sent by the relay to the origin.

Then there is indeed the cost of caching. A relay serving a "streaming" media may or may not have resource to cache the whole segment. If it does not, it maintains conceptually a "per client" cache of that segment -- and that cache may well be as limited as the cache of real time media. If a new client comes, that new client will not be served from the cache, but will get its own copy from the origin. Implementing that is a bot more complicated than "caching everything", so I am not sure we should support it in the prototype, at least not in the first version. Maybe something to consider later.

As for the cache logic, we really want the relay to be media-agnostic. We have information provided for congestion control purpose: priority of the object, and a can-drop bit. We can use these to implement the "catch up" logic. Start from the current GOB, drop everything that is marked can-drop until sync is achieved, skip immediately to a new GOB if it is ready. That gives us media-agnostic catch up, without requiring new data.

[suhasHere]

the unit of caching is always Group of objects. If a given cache stores one or more of it depends on several reasons - current cache depth, cdn caching policies, publisher mentioned TTL and so on.

So client can ask

• from the beginning of the current group
• from the current object
• starting from a given group/object

Output in all these cases are limited by what's in the cache at a given time.

For the interactive video, it is totally fine to send to client all the objects from the beginning of the group. Whether the client chooses to render/not-to-render/speed-up/slow-down is client's decision and shouldn't be enforced by the cache in my opinion. I still don't see a strong reason to mark source type.

It makes me think we need something that is like video FIR request where the client asks for fresh idr to resync or on joining new. This can be its own moq message that is forwarded to the origin and actual subscription (waitup) mode follows.

[huitema]

I am almost done implementing this, with three options:

• synch to current gob: start sending with the last received gob, from the beginning of that GOB
• synch to current object: start sending with the last received object
• synch to specified point: start sending at a specific GOB/Object, or at the beginning of the cache if later

We can then experiment.

[huitema]

Please open a separate issue for "on demand resynch".

[huitema]

Subscribe intent is almost ready in PR #94. Took me some time, because implementing it surfaced a couple of bugs, took me some time to identify and fix the issue. That's par for the course as we have been updating specifications on the fly.

We have discussed an option for "synch to current object". It is not hard to implement, but it is hard to test it in a robust way, because the "current object" depends on details of implementation, packet losses, etc. If I program the tests to check that the "current object" is group_id=1, object_id=15, these tests will work on my dev box, but they may well fail on linux, or fail in the future if we change some detail of the scheduling. I am tempted to implement that as "synch to next group", which will be in line with always aligning to the beginning of the group. This makes no difference for audio. For video, it will create a wait until the source sends the next block. (This can be mitigated later with "on demand resynch".)

[huitema]

PR #94 is ready. Implements subscribe intent, supporting three types of intent:

• current GOB
• next GOB
• specified start point. (First object available after specified point if specified point not available any more.)

This changes some of the API:

• For object source creation, addition of a property specifying whether source is "real time" or not. If source is real time, the cache is pruned continuously, to only keep the last relevant GOB.
• For media subscribe, addition of an intent parameter. If set to NULL, this is set by default to "specified start point 0, 0".

Version bump, so we do not mix several protocol versions in test.

[huitema]

I have checked in PR #94. I think this issue can be closed.