search

w3c / mediacapture-transform

MediaStreamTrack Insertable Media Processing using Streams
https://w3c.github.io/mediacapture-transform/
Other
44 stars 19 forks source link

Memory management for incoming frames #6

Closed youennf closed 3 years ago

youennf commented 3 years ago

In some cases, MediaStreamTrack might produce more frames than the JS pipeline might be able to consume. In that case, it seems good that frames get dropped, for instance only the last frame received by the source is actually enqueued.

MediaStreamTrack may have different sources: camera, canvas, network. Dropping frames might be ok with a camera since it will generate frames regularly. This might be an issue for other sources like canvas where dropping a frame is visible to JS, as well as remote tracks (with webrtc stats).

That begs the question to whether different implementations should behave the same in the way they handle cases where frames get enqueued quickly.

aboba commented 3 years ago

The "memory" in question can be a GPU buffer for some sources (e.g. zero copy from a camera to a GPU buffer).

alvestrand commented 3 years ago

The approach pioneered by WebCodec is that the frame has a destroy() method, and an API requirement that destroy() has to be called before the frame is released; if the frame is passed to a MediaStreamTrackGenerator, this will be handled by the generator.

The practical limit on outstanding frames in the Chrome implementation seems to be somewhere larger than 3, but less than 10.

guidou commented 3 years ago

I think the WebCodecs VideoFrame is quite flexible in this regard, as @alvestrand points out. cc @chcunningham of the WebCodecs team, who might be able to provide additional input.

chcunningham commented 3 years ago

For the case where apps are reading fast enough and simply want to drop some of the frames, Harald and Guido gave a good summary on how to release the frame. The method is now called close() (was destroy()). https://github.com/w3c/webrtc-encoded-transform/issues/99#issuecomment-824494217 has some additional details.

Also, for the case where apps are not reading fast enough, the MediaStreamTrackProcessorInit now includes a maxBufferSize argument that gives the app control over the frame dropping behavior. The queue lives inside the MediaStreamTrackProcessor and it's oldest item is dequeued to satisfy read requests from the ReadableStream. If the arrival of a new frame would cause the queue to exceed maxBufferSize, the oldest item in the queue is dropped prior to enqueueing the newest arrival.