search

w3c / mediacapture-transform

MediaStreamTrack Insertable Media Processing using Streams
https://w3c.github.io/mediacapture-transform/
Other
44 stars 20 forks source link

Playback and synchronization of MediaStreamTracks created by VideoTrack Generator #86

Open aboba opened 2 years ago

aboba commented 2 years ago

VideoTrackGenerator (the successor to MSTG) converts a stream of VideoFrames to a MediaStreamTrack.

VideoFrames may have been "transformed" by a process that generates jitter. For example, they may have been processed by a Machine Learning algorithm that takes a varying amount of time, or they may have been produced by a decoder that received encoded chunks over a network with varying delays.

Currently, the specification does not talk about what an application can expect of VideoTrackGenerator in terms of playout delay and rendering smoothness. Does VideoTrackGenerator assume that there is a jitter buffer upstream that generates VideoFrames with constant delays? Or does VideoTrackGenerator adjust the playout delay in response to the jitter it experiences in the VideoFrames it receives?

tonyherre commented 2 years ago

Should the VideoFrame's timestamp relate to this as well? The webcodecs spec defines it as "The presentation timestamp", with a note about potentially using it to guide rate control in the encoder. It would be great if we could specify its behaviour in the context of mediacapture transforms.

alvestrand commented 2 years ago

I think the timestamps in the stream coming from VTG are the timestamps coming into the VTG - dejittering should happen just before playback (or other process that needs a regular cadence).

Needs discussion.

dontcallmedom-bot commented 1 month ago

This issue was discussed in Joint Media/WebRTC WG meeting at TPAC – 26 September 2024 (Playback and sync of tracks created by VideoTrackGenerator)