search

w3c / mediacapture-transform

MediaStreamTrack Insertable Media Processing using Streams
https://w3c.github.io/mediacapture-transform/
Other
44 stars 19 forks source link

Effect of Jitter on VideoTrackGenerator #86

Open aboba opened 1 year ago

aboba commented 1 year ago

VideoTrackGenerator (the successor to MSTG) converts a stream of VideoFrames to a MediaStreamTrack.

VideoFrames may have been "transformed" by a process that generates jitter. For example, they may have been processed by a Machine Learning algorithm that takes a varying amount of time, or they may have been produced by a decoder that received encoded chunks over a network with varying delays.

Currently, the specification does not talk about what an application can expect of VideoTrackGenerator in terms of playout delay and rendering smoothness. Does VideoTrackGenerator assume that there is a jitter buffer upstream that generates VideoFrames with constant delays? Or does VideoTrackGenerator adjust the playout delay in response to the jitter it experiences in the VideoFrames it receives?

tonyherre commented 1 year ago

Should the VideoFrame's timestamp relate to this as well? The webcodecs spec defines it as "The presentation timestamp", with a note about potentially using it to guide rate control in the encoder. It would be great if we could specify its behaviour in the context of mediacapture transforms.

alvestrand commented 1 year ago

I think the timestamps in the stream coming from VTG are the timestamps coming into the VTG - dejittering should happen just before playback (or other process that needs a regular cadence).

Needs discussion.