Ability to tell MediaRecorder to maximize frame rate

[j0000el]

Videos produced via the MediaRecorder API have highly inconsistent frame rates. Using the same stream settings and MediaRecorder configuration, captured videos can range from average frame rates that are good (30ish, assuming stream frame rate is 30) to bad (below 20). While inconsistent frame rates could be acceptable in the real-time communication case, they are not acceptable if you're building a camera app that captures video with the Stream/MediaRecorder APIs.

The issue is most pronounced when using a mobile device (tests done in my case with Pixel 1 and 3 on Chrome). MediaRecorder seems to prefer delivering frames in real-time, so if encoding a frame takes too long, it simply drops it and moves on to the next. Since mobile devices are slower, potentially more likely to be overloaded, I believe the issue is much more pronounced there.

Rather than delivering frames as fast as possible, dropping frames if it can't keep up, I would like to see the ability to tell MediaRecorder to prefer a higher, more consistent frame rate on the encoded video blob, even if the final blob lags a bit after MediaRecorder.stop() is called (dataavailable won't be fired immediately, for instance, until encoding is complete).

I can see this as a new option that can be passed in the constructor. Passing a target frame rate is likely not a great option, as that would be dependent on the input stream frame rate and the browser should likely retain some flexibility to drop frames if the device is severely overloaded (even native camera apps drop frames, just not as bad as MediaRecorder on the same device). Perhaps just a hint to the encoder that more time should be dedicated to each frame before skipping to the next, potentially causing a lag in delivering video segments to the client. My naming-foo is not strong here, so open to suggestions, but something like a boolean property named 'videoMaximizeFrameRate' (default false).

const mediaRecorder = new MediaRecorder(stream, {
  mimeType: 'video/webm',
  videoMaximizeFrameRate: true,
});
mediaRecorder.start();
// Some time passes.
mediaRecorder.ondataavailable = () => {
  // Video blob delivered after some delay, with encoded frame rate as close to input
  // stream frame rate as possible.
};
mediaRecorder.stop();

[jesup]

Note that if the encoding lags, and you don't drop frames, you have to buffer video. For a short time and a small mismatch, this may be ok, but it rapidly becomes untenable especially on memory-constrained devices. If you want to prefer consistent framerate when recording a live source (as opposed to a pull-on-demand-from-a-file encode), you have to let something else bend, like downscaling the video. You could add a "prefer framerate over resolution" hint/control (framerate, resolution, auto(default) perhaps, or remove auto and make resolution the default to match current impls)

[alvestrand]

Isn't this going to be the same in practice as the "delay buffer adjustment" case? IE when you give the encoder more time to complete each frame (a larger delay buffer), you are also likely to get a more consistent frame rate. If the CPU capacity available isn't enough to encode the incoming data, something will break anyway, of course.

[j0000el]

I think adjustments to the delay buffer are likely the right approach for achieving more consistent frame rates without creating a potentially large buffer of video, though is that too low level to expose to the user? I was thinking the user would provide a hint of whether they want to prefer a consistent frame rate or fast frame delivery, and the browser could deploy whichever strategy to achieve that, so delay buffer adjustments would be more of an implementation detail. Or is the suggestion that this improvement can be achieved across the board, without a hint from the user?

I understand something has to give if we increase the buffering time. I would prefer delays in frame deliver or an increase (though reasonably capped) in memory usage due to buffering over lowering resolution automatically. If the goal is to produce a more high quality video, resolution and frame rate both matter. At the very least, I would want the user to be able to specify the minimum resolution acceptable if it gets auto downgraded to boost frame rate.

[sr1dh4r]

I was looking at this issue on mobile device as well.

From what I see, vp8 vs vp9 when specifying mimetype ("video/webm;codecs=vp8") seems to choose between max framerate vs max quality and with vp9, frame drop is huge on mobile device.

But this looks like its due to device limitation since even video playback in vp9 is affected (even with use of requestAnimationFrame) because we are encoding while playing the stream. I guess I need someway to record raw stream and then use that stream for offline encoding.

Seems like the timeslice option should just do that but I don't see it deferring the processing load.

[guest271314]

In theory canvas.captureStream(0) and <CanvasCaptureMediaStream|MediaStream>.requestFrame() and ImageCapture.grabFrame() and can be used to create video with the required frame rate, https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html, https://plnkr.co/edit/oYRSCpNQAL92Yb5o?preview.

[sr1dh4r]

In theory canvas.captureStream(0) and <CanvasCaptureMediaStream|MediaStream>.requestFrame() and ImageCapture.grabFrame() and can be used to create video with the required frame rate, https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html, https://plnkr.co/edit/oYRSCpNQAL92Yb5o?preview.

Canvas stream capture makes it worse even with vp8 encoding since it has to a lot more work to do.

Also, RequestFrame would involve running a loop and there is no way one can sync with the frame rate being provided without losing frames and/or affecting the UI performance.

[guest271314]

Canvas stream capture makes it worse even with vp8 encoding since it has to a lot more work to do.

Also, RequestFrame would involve running a loop and there is no way one can sync with the frame rate being provided without losing frames and/or affecting the UI performance.

What evidence lead you reach that conclusion?

It is possible to essentially set the exact number of frames using ImageCapture.grabFrame() and a TransformStream of asynchronous generator.

What specific frame rate are you expecting, consistently, that you have not been able to achieve?

30 frames per second? 60 frames per second?

What methodology did you use to verify how many frames are written to the resulting media file?

[guest271314]

Note that frame rate necessarily must change when input is variable width and height frames https://plnkr.co/edit/4Tb91b?preview, https://plnkr.co/edit/Axkb8s?preview. webm-writer which decodes WebP image VP8 frame has the ability to set frame rate https://plnkr.co/edit/Inb676?preview, in pertinent part, setting frame rate manually

      for (const blobURL of media) {
        await new Promise(async resolve => {
          video.addEventListener("pause", e => {
            controller.close();
            const currentFrames = frames[frames.length - 1];
            const [frame] = currentFrames;
            frame.duration = video.currentTime - frame.duration;
            frame.frameRate = (frame.duration * 60) / currentFrames.length -1;
            done.then(resolve);
          }, {
            once: true
          });
          video.src = blobURL;
        });
      }
      console.log(frames);
      for (const frame of frames) {
        const [{
          duration, frameRate, width, height
        }] = frame;
        console.log(frameRate);
        const framesLength = frame.length;
        const frameDuration = Math.ceil((duration * 1000) / framesLength);
        for (let i = 1; i < framesLength; i++) {
          videoWriter.addFrame(frame[i], frameDuration, width, height);
        }
      }

MediaRecorder could add that option. The requirement is possible now using various means, include pause(), resume(); seeking to every frame https://bugs.chromium.org/p/chromium/issues/detail?id=1065675.

[ivictbor]

On a new laptop I had frame rate near 15-20 FPS after MediaRecorder when used the next constrainsts:

await navigator.mediaDevices.getUserMedia({
      video: {
        frameRate: {
          ideal: 24,
          max: 25,
        },
      },
});

On old laptop they produced 24 FPS.

Fixed when I added min constraint:

await navigator.mediaDevices.getUserMedia({
      video: {
        frameRate: {
          min: 24,  // 😲 this constraint fixed an issue
          ideal: 24,
          max: 25,
        },
      },
});

Important Note from https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/MediaRecorder

Video resolution, frame rate and similar settings are specified as constraints when calling getUserMedia(), not here in the MediaRecorder API.

[alvestrand]

@ivictbor please open a new issue when you want to discuss a new topic; this issue was related to MediaRecorder, while your issue was related to getUserMedia. FWIW: When you ask for "ideal: 24, max: 25", you are asking for a frame rate no higher than 25 and as close to 24 as the platform finds reasonable, but you're happy with any limit. You're likely seeing the camera choose the closest native frame rate (likely 20) in the first case, and a higher frame rate (likely 50) winnowed down to 25 in the second case. Not possible to say more without enumerating the modes available on your camera - this would be a browser issue, not a spec issue.

[ivictbor]

@alvestrand, thanks, I had no issues, just shared information if someone else is looking for it, because when I Googled initial question, I found this thread, so if anemone else will do it, probably my comment will help