Webm plays way too fast when encoded on iOs 16.4

[davidstrahm]

Hello there Not sure if it has something to do with webm-muxer, which is awesome by the way.

I‘m using your library with VideoFrame from a live webrtc feed. I‘m encoding it with vp8 and vp9 codec using WebCodec API. In the next version of safari (16.4) WebCodec API is also available in safari.

It works great on android, but when i create webm videos from safari, the resulting videos play very fast. Also i cannot open it in VLC, it plays in windows native player though.

I tried reducing the framerate that the VideoFrames are pushed to the encoder so that the result is about 15fps, again working as expected on Android. Do you have any hints regarding this? As i understand, setting the fps param in webmmuxer is only informative (metadata) right? Because i send each frame manually, I dont think the fps param of VideoEncoder has any influence.

[Vanilagy]

Thank you!

Could you send the faulty WebM file that you encoded? Would be helpful 'cause I could inspect it. And yes, you are correct; the frameRate parameter for WebMMuxer is for metadata only. Have you tried inspecting, i.e. console.logging the VideoFrames' timestamps? Those ultimately determine the playback rate of the video! If you notice anything fucky with those, you can always manually override the timestamps when passing the encoded chunks into the muxer (more on that in the README.)

[davidstrahm]

Thank you for getting back I have inspected the frame timestamps, but have not overwritten the timestamp parameter yet because the timestamps seemed ok (it was the default long timestamp from the videostream, not beginning at 0, i used the offset parameter). Will definitely try this. What tool do you use to inspect the webm files? I tried mkvalidator and got some errors. but then again i got the same errors on a working file.

Meanwhile, here is a webm from iphone 12. Also, the aspect ratio is wrong (is correct on android with the same settings). iphone12_2.webm

[Vanilagy]

That file indeed looks fucked. Can you send me the relevant code, i.e. the code where you create the muxer, the encoder, where you encode the frames and then ultimately pass them into the muxer? That would help me see if this is an error in your usage or a bug in the library. If you say the same code produces correct files on Android but bullshit on iOS, that is indeed sus. Let's hope this is not a Safari bug!

I use MKVToolNix to inspect my WebM files.

[davidstrahm]

sure, here is the relevant code. yes, it works on android. my current theory is that maybe the timestamp in safari video stream is different than on android, but i will check it next week and try to set the timestamp manually. wouldn't be the first safari bug (sigh)

 // in main thread
  // main animationframe loop, receives frames as fast as possible
  interval(0, animationFrameScheduler).pipe(
      observeOn(asyncScheduler),
      throttleTime(66), // throttles frames to 1000/15=66ms = ~15fps
      exhaustMap(() => {
          const videoFrame = new VideoFrame(_video); // _video is a HTMLVideoElement

          // send frame to encoder worker, take a new frame only if the previous frame was encoded
          // ...
          postMessage(EncodeWorkerRequest.ENCODE_FRAME({ frame: videoFrame }));

          // ... send stop message when video was stopped <omitted>
      }),
  ),

 // in worker: 
const encoderConfig = {
    codec: 'vp09.00.10.08',
    hardwareAcceleration: 'prefer-hardware',
    width: videoWidth,
    height: videoHeight,
    bitrate: 2_000_000,
    framerate: 15,
};

const muxer = new WebMMuxer({
    target: 'buffer',
    video: {
        codec: 'V_VP9', 
        width: videoWidth,
        height: videoHeight,
    },
    firstTimestampBehavior: 'offset', 
});

const encoder = new VideoEncoder({
    output: (chunk, meta) => {
        muxer.addVideoChunk(chunk, meta);
        // send message to main thread so that a new frame is sent
        postMessage(EncodeWorkerResponse.FRAME_ENCODED({})); 
    },
    error: e => console.error(e),
});
encoder.configure(encoderConfig);

// when done:
await encoder.flush();
encoder?.close();

let buffer = muxer.finalize();
if (buffer) {
    // send finished webm buffer back to main thread
    postMessage(EncodeWorkerResponse.ENCODING_COMPLETE({ data: buffer }));
);

[Vanilagy]

Yeah, the code looks normal, nothing strange I see in it. Are you dealing with real-time data (like the data coming from a media stream, such as a camera), or with pre-recorded data? If you're dealing with the latter, the throttleTime(66) seems at least a bit strange.

If it's a timestamp thing, which you can find out easily by logging them, that would show a difference between Safari and Chromium.

[davidstrahm]

I figured it out. Timestamps in Safari are always 0 🙄. It works now by overriding, so it has nothing to do with webmm-muxer Thanks for your help

[Vanilagy]

Perfect! 😭