Timing accuracy and decode/presentation ordering

[chrisn]

In the W3C meeting on 2022-01-17 @ytakio asked whether SEI events should be triggered in the browser before or after decoding? The decode order of video frames may differ from presentation order.

We should also clarify the timing requirements for firing the event messages based on use cases. Is the intent to perform frame accurate rendering of overlay content (e.g., in the bullet chat use case). In such scenarios, I wonder if exposing SEI events through WebCodecs is more appropriate, because synchronization between DOM updates and media playback can be difficult.

[leonardoFu]

I think SEI event should be triggered before decoding, because browser only pre-decode several frames to reduce the memory usage. As to the accuracy, I think in most of our use cases, 100 millisecond is good enough. If you really need to synchronize SEI with video frame perfectly, you can use the WebCodecs and Video.requestVideoFrameCallback @ytakio @chrisn

[ytakio]

Sorry for my late reply... Yes, I agree that it would be better to be triggered before decode (at parsing). On the other hands, @chrisn already mentioned above, I wonder if web developer and DOM rendering can be acceptable to handle the decoding order which may have some reversed timestamp. So I think that it should be compared and considered for the case where events are fired after timing of output from decoded picture buffer (this buffer must be included in each decoder in general), too.

[nigelmegitt]

I wonder if web developer and DOM rendering can be acceptable to handle the decoding order which may have some reversed timestamp.

This was my concern as well - out of order timestamps are possibly much worse for the client to deal with than in-order-but-a-bit-late timestamps. I'm not sure what all the use cases are though, and if this could be a real world issue.

[chrisn]

We discussed in the 2022-02-21 meeting that the WebCodecs spec requires VideoFrames to be emitted in presentation order. If we do need to expose SEI events in decode order, we should update https://github.com/w3c/webcodecs/issues/198 to describe this.

Another consideration is how far ahead an application would need to see the SEI event to be able to render using information in the event. Could this be handled by application-level buffering of VideoFrames, or should the WebCodecs API somehow expose SEI events before the VideoFrame?