search

w3c / mediacapture-transform

MediaStreamTrack Insertable Media Processing using Streams
https://w3c.github.io/mediacapture-transform/
Other
44 stars 20 forks source link

Mediacapture-transform Timing Model #88

Open aboba opened 1 year ago

aboba commented 1 year ago

This is an overall tracking issue for timing-related issues in mediacapture-transform.

Questions:

  1. What timing information is provided by various sources (capture, canvas, etc.)
  2. What VideoFrame/AudioData attributes correspond to that timing information?
  3. What timing attributes are maintained between Video/AudioFrames and encodedChunks (encoder) and between encodedChunks and Video/AudioFrames (decoder)?
  4. What is done with this timing information in the Web platform? For example, how are MSTs sync'd within a MediaStream?

Timing Model discussed at November VI

VideoFrameCallbackMetaData in rVFC specification includes receiveTime, captureTime, rtpTimestamp, mediaTime.

Resolution: “file specific issues on specific specs”

Related: Issue 601: Expose in VideoFrame

Use case: https://www.w3.org/TR/webrtc-nv-use-cases/#auction

aboba commented 11 hours ago

Timing-related issues:

80 Expectations/requirements for VideoFrame and AudioData timestamps

86 Playback and synchronization of MediaStreamTracks created by VideoTrack Generator

87 What is the timestamp value of the VideoFrame and AudioFrame from a remote webrtc stream track?

96 What is the impact of timestamp for video frames enqueued in VideoTrackGenerator?