MediaStreamTrackGenerator/MediaStream should buffer the current frame

[jamiebuilds]

Right now, at least in Chromium's implementation which they say is to spec:

If you write a frame to MediaStreamTrackGenerator and then assign its container MediaStream to a video.srcObject you will never see that frame.

let track = new MediaStreamTrackGenerator({ kind: "video" });
let stream = new MediaStream([track]);
let writer = track.writable.getWriter();

// First write frame:
await writer.write(videoFrame);
// Then assign:
videoElement.srcObject = stream

// Result: Frame never appears

This is different than the behavior of a MediaStream created by HTMLCanvasElement.captureStream()

let stream = canvas.captureStream(0) // manually request frames
let track = stream.getVideoTracks().at(0)

// First write frame:
track.requestFrame();
// Then assign:
videoElement.srcObject = stream

// Result: Frame appears

Besides the behavior being different, it also makes using these streams inconvenient for using across multiple <video> elements which is common in a lot of video calling apps, and you end up needing to hold onto the current frame anyways.

// This will cause frames to be dropped every time you switch views/speakers/etc:
function onInAppPictureInPicture() {
  let video = document.createElement("video")
  video.srcObject = getCurrentSpeakerStream()

    function onCurrentSpeakerChange() {
      video.srcObject = getCurrentSpeakerStream()
    }

  // ...
}

Note: This can be very noticeable in realtime calling apps because often the next frame can take awhile to appear, and in the meantime you just see a blank video feed instead of the last frame.

Instead, if you want to avoid recreating the same stream again, you would have to hold onto the last frame and write it to the stream again when necessary:

let track = new MediaStreamTrackGenerator({ kind: "video" })
let writer = this.#trackGenerator.writable.getWriter()
let lastFrame

async function writeFrame(frame) {
  lastFrame = frame
  await writer.write(frame)
}

async function replayLastFrame() {
  await writer.write(lastFrame)
}

This difference between CanvasCaptureMediaStreamTrack vs MediaStreamTrackGenerator caused a real regression in our app when attempting to use it.

[youennf]

track.requestFrame(); will trigger a frame emition but the frame emition might happen asynchronously (even in a different thread). Hence why it might be rendered.

Please also note that, as per spec, video frames would be enqueued in a worker. So you would need to ensure srcObject is set in window before enqueueing video frames in the worker (through postMessage).

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC November 19 2024 meeting – 19 November 2024 (Issue #114: VideoTrackGenerator/MediaStream should buffer the current frame):

RESOLUTION: Not buffering the last frame is the expected behavior