Closed youennf closed 3 years ago
The "memory" in question can be a GPU buffer for some sources (e.g. zero copy from a camera to a GPU buffer).
The approach pioneered by WebCodec is that the frame has a destroy() method, and an API requirement that destroy() has to be called before the frame is released; if the frame is passed to a MediaStreamTrackGenerator, this will be handled by the generator.
The practical limit on outstanding frames in the Chrome implementation seems to be somewhere larger than 3, but less than 10.
I think the WebCodecs VideoFrame is quite flexible in this regard, as @alvestrand points out. cc @chcunningham of the WebCodecs team, who might be able to provide additional input.
For the case where apps are reading fast enough and simply want to drop some of the frames, Harald and Guido gave a good summary on how to release the frame. The method is now called close() (was destroy()). https://github.com/w3c/webrtc-encoded-transform/issues/99#issuecomment-824494217 has some additional details.
Also, for the case where apps are not reading fast enough, the MediaStreamTrackProcessorInit now includes a maxBufferSize argument that gives the app control over the frame dropping behavior. The queue lives inside the MediaStreamTrackProcessor and it's oldest item is dequeued to satisfy read requests from the ReadableStream. If the arrival of a new frame would cause the queue to exceed maxBufferSize, the oldest item in the queue is dropped prior to enqueueing the newest arrival.
In some cases, MediaStreamTrack might produce more frames than the JS pipeline might be able to consume. In that case, it seems good that frames get dropped, for instance only the last frame received by the source is actually enqueued.
MediaStreamTrack may have different sources: camera, canvas, network. Dropping frames might be ok with a camera since it will generate frames regularly. This might be an issue for other sources like canvas where dropping a frame is visible to JS, as well as remote tracks (with webrtc stats).
That begs the question to whether different implementations should behave the same in the way they handle cases where frames get enqueued quickly.