Is RTCEncodedVideoFrameMetadata.frame_id actually an unsigned long long or does it wrap at 16 bits?

[tonyherre]

frame_id is defined as a unsigned long long, but currently doesn't have any textual definition in the members section beneath. Same for dependencies.

My rough understanding is that it corresponds to the Dependency Descriptor Header Extension's frame_number field as a codec-agnostic frame numbering method, but that field is only 16 bits, so is the UA assumed to be handling the 16 bit wrapping and converting up to a 64 bit unsigned long long? That's the current Chromium implementation at least. I'm unsure of other browsers.

[aboba]

@tonyherre I had the same question. Is it likely you'd have a dependency that would require the full 64 bits of an unsigned long long??

Now that Eugene is looking at implementing frame_id and dependencies in WebCodecs, it would be nice to get this figured out.

[fippo]

We had a somewhat similar discussion for the inbound rtpTimestamp in #154

If the field is present both outbound and inbound we should aim for consistency even if that means expanding to the long form.

One problem of the short form is that it may be codec dependent, e.g. VP8 has a 15 bit field (and a 7 bit one but lets pretend it does not exist...) so how would we expose the limit?

[tonyherre]

If it might only be using 15 or 16 bits, then a web app doing anything with managing frame dependencies (broadcast with fanout, dropping layers etc) would have to handle wraps after 18/36 minutes of 30fps which is a pain. 32 bits would be enough I'd think, if someone's worried about size.

I'd suggest we leave the type as-is, and just add definitions along the lines of:

frameId: A monotonically increasing frame counter. Its lower 16 bits will match the frame_number of the Dependency Descriptor Header Extension, if present.

dependencies: List of the frameIds of RTCEncodedVideoFrames required to decode this frame.

WDYT?

[alvestrand]

WFM, but need to have language that it matches the frame ID in the encoded frame in its lower bits when that exists. (Is the DD frame ID defined to be 16 bits? How does it match up with VP8's 7-bit frame ID variant?)

[fippo]

On the receive side we might make this depend on the presence of DD to avoid codec-specific wrapping (the 7bit VP8 variant is broken anyway but the normal one would be 15 bits not 16)?

[aboba]

Do we understand how frameId is supposed to behave across codecs? For WebCodecs PR 756, we are going to need to specify how it is initialized, incremented, etc.

[aboba]

@tonyherre dependencies is the list of frames that the current frame references. It is not the complete list of frameIds necessary to enable the frame to be decoded. Since a non-keyframe dependency will itself have dependencies, even if a receiver has received all the dependencies it still may not be able to decode the frame if the dependency chain has been broken. This is why DD includes chains as well as dependencies.