Add "real-time" warning/note to VideoTrackGenerator

[tguilbert-google]

MediaStreamTrackGenerator does not specifiy any limits as to the speed at which data can be fed in. This can lead to problems if developers push 60s worth of audio data in 1s (see crbug.com/1184070).

There should be a note in the spec mentioning that developers are expected to feed tracks at a reasonable real-time rate.

[guidou]

We should indeed add this note to the spec. Some mechanisms that could be used to notify applications about this type of problem:

• Platform sinks can use existing error-reporting mechanisms (e.g., error event for media elements and maybe peer connectionss)
• We can define a signal that can be exposed via MediaStreamTrackProcessor.readableControl

[guest271314]

This issue is still ripe. A note should be a sooner rather than later. An author of the WebCodecs specification author filed a Chromium bug due to the expectation that a MediaStreamTrackGenerator in a MediaStream set as srcObject of an <audio> element could be written to in a synchronous for loop without delay between calls and the audio would be output at the element at the same pace as delaying writes between calls to write() https://bugs.chromium.org/p/chromium/issues/detail?id=1260519.

[alvestrand]

the input to the MSTG is a Stream. I would suggest that the note should say that the client of MSTG (or now: VideoTrackGenerator) needs to respect the high water mark of the stream, otherwise frames may be lost, and that one should expect the MSTG to process frames at a rate no slower than real time, but not much faster either.

[dontcallmedom-bot]

This issue was discussed in WebRTC November 19 2024 meeting – 19 November 2024 (Issue #20: Add “real-time” warning/note to MediaStreamTrackGenerator)