Inconsistent behavior between WebMMuxer and Mp4Muxer

[cdaein]

Hi, Thank you for all the work on webm-muxer and mp4-muxer!

I am using the libraries to record canvas animation frames. I've noticed each library behaves a little differently when used in the same way.

In the example below, I'm recording the total of 30 frames.

In the resulting WebM video, the last frameCount is 29, which is accurate. It also has the correct fps of 30.

But in the mp4 video, the last frameCount shows up as 28, missing the very last frame. What's also strange is that when I manually scrub the Quicktime timeline, I sometimes do see 29 displayed but it seems this frame doesn't have any duration. Also, the frame rate is displayed as 31.03 instead of 30.

I'm wondering if I need to handle frame recording differently with mp4-muxer.

Below is a simple demo to illustrate my issue. You can click the mouse inside the canvas to initiate the recording of both videos:

import * as Mp4Muxer from "mp4-muxer";
import * as WebMMuxer from "webm-muxer";

const canvas = document.createElement("canvas");
document.body.appendChild(canvas);
canvas.width = 400;
canvas.height = 400;
const ctx = canvas.getContext("2d")!;

let mp4Muxer: Mp4Muxer.Muxer<Mp4Muxer.ArrayBufferTarget> | null = null;
let webmMuxer: WebMMuxer.Muxer<WebMMuxer.ArrayBufferTarget> | null = null;
let mp4VideoEncoder: VideoEncoder | null = null;
let webmVideoEncoder: VideoEncoder | null = null;
let startTime: number | null = 0;
let recordingState: "start" | "recording" | null = null;
let lastKeyFrame = -Infinity;
let frameCount = 0;
let fps = 30;

const startRecording = async () => {
  mp4Muxer = new Mp4Muxer.Muxer({
    target: new Mp4Muxer.ArrayBufferTarget(),
    video: {
      codec: "avc",
      width: canvas.width,
      height: canvas.height,
    },
    fastStart: "in-memory",
  });
  webmMuxer = new WebMMuxer.Muxer({
    target: new WebMMuxer.ArrayBufferTarget(),
    video: {
      codec: "V_VP9",
      width: canvas.width,
      height: canvas.height,
      frameRate: 30,
    },
  });

  mp4VideoEncoder = new VideoEncoder({
    output: (chunk, meta) => mp4Muxer!.addVideoChunk(chunk, meta),
    error: (e) => console.error(e),
  });
  mp4VideoEncoder.configure({
    codec: "avc1.42001f",
    width: canvas.width,
    height: canvas.height,
    bitrate: 1e6,
  });

  webmVideoEncoder = new VideoEncoder({
    output: (chunk, meta) => webmMuxer!.addVideoChunk(chunk, meta),
    error: (e) => console.error(e),
  });
  webmVideoEncoder.configure({
    codec: "vp09.00.10.08",
    width: canvas.width,
    height: canvas.height,
    bitrate: 1e6,
  });
};

const encodeVideoFrame = () => {
  let elapsedTime = (frameCount * 1e6) / fps;
  let frame = new VideoFrame(canvas, {
    timestamp: (frameCount * 1e6) / fps, // Ensure equally-spaced frames every 1/30th of a second
  });

  // Ensure a video key frame at least every 10 seconds for good scrubbing
  let needsKeyFrame = elapsedTime - lastKeyFrame >= 10000;
  if (needsKeyFrame) lastKeyFrame = elapsedTime;

  mp4VideoEncoder?.encode(frame, { keyFrame: needsKeyFrame });
  webmVideoEncoder?.encode(frame, { keyFrame: needsKeyFrame });
  frame.close();
};

const endRecording = async () => {
  recordingState = null;

  await mp4VideoEncoder?.flush();
  mp4Muxer?.finalize();
  let mp4Buffer = mp4Muxer?.target.buffer!;
  downloadBlob(new Blob([mp4Buffer]), "test.mp4");

  await webmVideoEncoder?.flush();
  webmMuxer?.finalize();
  let webmBuffer = webmMuxer?.target.buffer!;
  downloadBlob(new Blob([webmBuffer]), "test.webm");

  mp4VideoEncoder = null;
  mp4Muxer = null;
  webmVideoEncoder = null;
  webmMuxer = null;
  startTime = null;
};

const downloadBlob = (blob: Blob, filename: string) => {
  let url = window.URL.createObjectURL(blob);
  let a = document.createElement("a");
  a.style.display = "none";
  a.href = url;
  a.download = filename;
  document.body.appendChild(a);
  a.click();
  window.URL.revokeObjectURL(url);
};

canvas.addEventListener("click", () => {
  recordingState = "start";
});

function run() {
  if (recordingState === "start") {
    frameCount = 0;
    startRecording();
    recordingState = "recording";
  }

  draw();

  if (recordingState === "recording") {
    encodeVideoFrame();
  }

  frameCount++;

  if (recordingState === "recording" && frameCount === 30) {
    endRecording();
    frameCount = 0;
    recordingState = null;
  }

  window.requestAnimationFrame(run);
}
window.requestAnimationFrame(run);

function draw() {
  ctx.fillStyle = "gray";
  ctx.fillRect(0, 0, canvas.width, canvas.height);

  ctx.font = "100px monospace";
  ctx.fillStyle = `white`;
  ctx.fillText(frameCount.toString(), 200, 200);
}

[Vanilagy]

Thanks for the full reproduction, I really appreciate it! Even better would be a finished setup on StackBlitz (with Vite or something), so I can just open it and see it immediately. But that's for next time :)

MP4 and WebM work a little different; in WebM, video frames only have a timestamp. In MP4, the timing of frames is determined by the delta between frames, so duration matters more. A simple fix in your code made the MP4 file play back correctly:

  let frame = new VideoFrame(canvas, {
    timestamp: (frameCount * 1e6) / fps, // Ensure equally-spaced frames every 1/30th of a second
+   duration: 1e6 / fps,
  });

Without this field, the frame implicitly gets a duration of 0. With the proper duration set, the last frame of the video shows correctly (this is in Finder):

https://github.com/Vanilagy/mp4-muxer/assets/1696106/d358ae9f-34e5-4ad2-81ae-3b79bb2b47fb

It has to be noted that this also depends heavily on the player, Chromium properly showed the last frame even for the "incorrect" file.

The 31.03 frame rate can be explained like so: Since the original video had 0-duration frames, its duration was equal to the timestamp of the last frame, which is at $\frac{29}{30}$ seconds. But the video still had 30 frames, so it comes out to a frame rate of

$$ \frac{30}{\frac{29}{30}} \approx 31.03448 $$

With the frame duration set, it correctly has a duration of exactly 1 second and a frame rate of 30.

I hope this clears things up. If you still have any questions, feel free to ask!

[cdaein]

That solved the issue! Thank you for the kind explanation. I will be keep in mind to include a live environment next time. 🙏

[Pensarfeo]

@Vanilagy, just to clarify? fixing the duration solves the problem on all players? specially interested in browsers

[Vanilagy]

@Pensarfeo It was the case for me, but I only tested Chromium and QuickTime. I attached a fixed file in my reply in this thread, so you can try that out.

[Pensarfeo]

Ill check! :)

[nuthinking]

Worked for me, thanks @Vanilagy!

[Vanilagy]

Awesome! I think this is partially on me since I don't set the duration in my demo script. I should!