Video is flickering, seems to be related to slow Android device

[rikschennink]

Hi,

Tried to use the muxer on a slower Android device and the resulting video flickers, almost seems like every second a frame is displayed from a couple milliseconds ago.

The video has a 1080 × 1920 resolution and is MPEG-4 AAC H.264 encoded.

Tried if videoBitrate had an effect but didn't seem to make any difference. Encoding on MacOS works correctly. This is with V_VP9 and vp09.00.10.08

I'm sorry I don't have any more specific details. Is there any reason this might happen? Anything I can configure to improve the output?

[kedicesur]

What is the original source for the videoFrames and what encoder are you using? I suspect we have to encode the received videoFrames with the same frameRate (fps) that they had been captured to eliminate their timestaps conflicting with the new encoding's framerate. However the JS Web API VideoFrame doesn't provide fps information but duration. In my project i implemented a 60 frames running average to resolve the fps from duration information and round the result. This yields 25, 30 50 or 60 and you best use it in your encoder's configuration as the framerate parameter.

[Vanilagy]

@rikschennink Where are you getting your VideoFrames from? The cool thing about WebCodecs, and the reason I created these libraries, is for rendering videos "offline" - meaning you can take as much time per frame as you want. Offline rendering basically means rendering duration != playback duration. This works exceptionally well when the frames come from some non-real-time process (like a canvas), but you can still make it work when it's coming from a webcam or something. The encoder will just "lag behind" the incoming frame stream. But since you're the PQINA guy, I assume it's the former for you!

@kedicesur "Frame rate" is automatically determined based on the timestamps of the video frames. Videos don't need to have a constant frame rate! But it's often desired, and easily achievable by just setting the timestamp manually on the VideoFrame based on some formula or an increment.

[rikschennink]

@Vanilagy Appreciate the quick response 🙏

I'll return with some more details and input / output video tomorrow morning (it's probably just me messing up).

[Vanilagy]

All good :)

[rikschennink]

I think it's probably caused by my own code seeking a position in the video and then drawing the video and creating the VideoFrame

I've attached the input and output video to give some context to the issue.

I'll close it for now, and will re-open if I feel the issue might be in the Muxer.

android-video-in.webm

android-video-out.webm

[rikschennink]

This is caused by my own doing, something wrong with the code that did video seeking, if I wait a couple milliseconds after onseeked the encoding works correctly.