Frames are being dropped, not smooth animation

[hills]

• I'm submitting a ... [x] bug report [ ] feature request [ ] question about the decisions made in the repository [ ] question about how to use this project

• Summary

The main page is clear that:

no frames are missed during video capturing

but this does not seem to be happening in practice.

I was hoping to smooth (guaranteed) videos of a series of web pages, by having them rendered offline one frame at a time, not realtime.

Happens in both the 'old' and 'new' headless modes.

• Other information (e.g. detailed explanation, stack traces, related issues, suggestions how to fix, links for us to have context, eg. StackOverflow, personal fork, etc.)

See the attached example, where frames 7-54 are identical and then the animation skips forward on frame 55. Using a low-quality setting to the codec it can be seen that the same pixels being handed to the codec as multiple frames. And in general this stutter continue across the rest of the recording.

https://user-images.githubusercontent.com/1131584/234968083-9d7a945c-7fb7-4e37-b1ad-682ce69c2079.mp4

$ cat /etc/alpine-release     # Alpine Linux
3.16.5

$ env PUPPETEER_CACHE_DIR=$(pwd) PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser node ./test.js

const puppeteer = require('puppeteer');
const { PuppeteerScreenRecorder } = require('puppeteer-screen-recorder');

const Config = {
    fps: 60,
    videoCrf: 28,
};

(async () => {
//  const browser = await puppeteer.launch({headless: 'new'});
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    page.setViewport({width: 1920, height: 1080});
    const recorder = new PuppeteerScreenRecorder(page, Config);
    await page.goto('https://developer.mozilla.org/en-US/docs/Web/CSS/animation');
    await recorder.start('recorded.mp4');
    await later(5000);
    await recorder.stop();
    await browser.close();
})();

function later(delay) {
    return new Promise(resolve => setTimeout(resolve, delay));
}

[Ministorm3]

I believe there are two issues at play here (although I am still investigating myself).

First, I believe #58 needs to be implemented. When I added that to my custom build, the animation is alot smoother, but too fast.

Second, I think there is an issue with FFmpeg being inside the same loop as whatever is putting together the frames for the video. Since FFmpeg is forced to operate at the same speed as the FPS the website can effectively render, the page appears to speed up.

For example, let's say the rendering is moving at 10fps, but FFmpeg is set for 30fps. FFmpeg receives 30 frames over 3 real seconds, but the video is setup to encode at 30fps. As a result, the video appears to be smooth, but moving at 3 times the speed.

I believe the solution is two parts for this loop issue. First, add -vf to the ffmpeg options so that it will continue to utilize the previous frame if the next frame is not available yet. But, since FFmpeg is still within this loop, it will still render at the same speed as the rendering. So, to resolve that, second, we need to move FFmpeg to its own worker thread so it can operate at maximum speed, even if the rendering is much slower. It will result in a choppier video (if your system can't render at maximum speed), but the video will render in real time. One second of video, even if only 10 real frames, would produce a full one second, and 30 frames of video.

This is a pretty complex solution, and currently I am speculating as to the exact issue at this point. I am continuing to dig into the issue myself, but this definitely isn't my full time job. So, someone smarter than me can probably tell me if I am on the right track.

My second point might still be valid. But it does not actually help what you are trying to do here.

[Ministorm3]

Oh also, if you are looking for a frame perfect rendering, you can also try https://github.com/tungs/timecut

[mattpr]

timecut rendered even worse for me (very long white screen and then 10 second of animations crammed in last half second and then the video ends). Definitely not representative of what you see in the browser directly, not even close. The initial result using suggested commands from the README were so bad I didn't bother messing around with it further.

With puppeteer-screen-recorder...

I noticed that if I don't set fps to 60 that I get a slow motion effect especially for any embedded video. Mentioned in other tickets.

The dropped frame problem (for me) turned out to be lack of hardware acceleration in headless. On mac I enabled this by passing chrome the following flags when running in headless...

            '--enable-gpu',
            '--use-angle',

You can use this trick to see if you successfully enabled your GPU in headless mode:

'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' --headless --enable-gpu --use-angle --print-to-pdf=out.pdf 'chrome://gpu'

Or try capture not in headless mode and see if it is smoother.

When switching to non-headless, the viewport wasn't working right for me: video smooshed/distorted and wrong final size (although everything looked fine in the visible window). Everything was fine in headless mode and the only thing I did was switch off the headless flag to chrome. So maybe a bug there or limitation of having a visible window.

[Vinlic]

I think my solution may have solved this problem. @hills @Ministorm3 @mattpr Please pay attention to my latest repo: https://github.com/Vinlic/WebVideoCreator It is currently in the process of improvement and is expected to complete all functions and usage documents within two weeks. If you have any questions, you can discuss with me.

You can verify it by importing index.js from the repository and using the following code：

import { examples, VIDEO_CODEC } from "./index.js"; await examples.synthesize({ url: "https://developer.mozilla.org/en-US/docs/Web/CSS/animation", width: 1280, height: 1280, fps: 60, browserUseGPU: true, videoCodec: VIDEO_CODEC.NVIDIA.H264, // You can choose an acceleration encoder based on your device outputPath: "./demo.mp4" });

https://github.com/prasanaworld/puppeteer-screen-recorder/assets/20235341/8bbf1d1a-566c-4dc2-995f-b2ed05b0d60d

[tansanDOTeth]

'--enable-gpu', '--use-angle',

Where do I pass in these options for the recorder?

[aske-cph]

Thanks for a great package!

Unfortunately i'm still seeing dropped frames in the new version (3).

See them both via the video in headless and non headless, even the though perfomance is "silky smooth" when observed via chrome (non headless), so it happens in the video capture.

I'm on a Macbook m2 (arm), and running chrome arm, so the correct version, ffmpeg is also arm, i have also tried various flags: there's also a link to more info on these here from browserless:

const browser = await puppeteer.launch({
                                               headless: true,
                                               args: [
                                                   '--use-gl=angle', // Use ANGLE for graphics rendering
                                                   '--use-angle=gl', // Choose OpenGL (GL) as the graphics backend for
                                                   '--enable-unsafe-webgpu' // https://www.browserless.io/blog/2023/08/31/browserless-gpu-instances/
                                               ]
                                           });

I've also tried fps 30/60/120 etc, to no avail.

A weird clue is that it seems to happen more on large resolutions or high speeds, and at more processor intensive animations - if i capture at mobile size there's almost no choppiness, and i if i scroll the page slowly with JS there's little frames dropped compared to if i speed it up. So it seems CPU/GPU load related, but only for the video capture as watching puppeteer live on chrome it's smooth.

[mattpr]

I've come to the conclusion that this is an issue with the screenshot api in terms of screenshot timing not giving enough frames or at the right time in order to maintain smooth animation with correct timing without any jerky/jumps.

I've tried various things like slowing down browser animation speed (to be re-timed after capture) so I get more frames over a given animation and any jumps in timing/frames are less noticeable...but didn't really make a difference....and not clear it is a reliable solution anyway if what you are capturing is a mix of video and js and css animations.

I think the best approach if you need smooth animation capture is to ditch the screenshot api which this project uses and do the following...

• use puppeteer to start/drive the browser to load/start/stop the pages you want to capture
• capture frames from a virtual monitor (frame buffer)

[kabapy]

I've come to the conclusion that this is an issue with the screenshot api in terms of screenshot timing not giving enough frames or at the right time in order to maintain smooth animation with correct timing without any jerky/jumps.

I've tried various things like slowing down browser animation speed (to be re-timed after capture) so I get more frames over a given animation and any jumps in timing/frames are less noticeable...but didn't really make a difference....and not clear it is a reliable solution anyway if what you are capturing is a mix of video and js and css animations.

I think the best approach if you need smooth animation capture is to ditch the screenshot api which this project uses and do the following...

• use puppeteer to start/drive the browser to load/start/stop the pages you want to capture
• capture frames from a virtual monitor (frame buffer)

After struggling with this issue, i think that processor speed is a critical factor. Agree with you. Any node package you recommend to capture from a virtual monitor?

[mattpr]

Don't have a package recommendation as I also haven't had much time to play with chrome -> Xvfb. I'm guessing you will need to connect together puppeteer (to drive the browser) with some sort of ffmpeg wrapper (for final encoding/retiming) and some kind of interface to Xvfb.

I also had the thought the other day that there might be enough screenshots...but that the lack of a timestamp on the screenshot when received means you don't know where it belongs on the timeline. You could get a batch of screenshots in a small interval of time but they may belong to different times (when captured vs when you get the data). Unless chrome adds some additional data (like screenshot capture timestamp) I think using the screenshot api for video capture will always be error prone because it isn't a "realtime API".

Yes you might have better results with certain hardware (processor, gpu, etc) and with certain webpages/animations... but the lack of ability to get predictable results makes using this API a no-go for video for anything serious where you need predictable results.

Google Web Designer app has a html5 to video rendering that comes out very good. it is an electron app, so you could maybe poke around in there to see if you can get some hints about how they are doing it. Maybe there is another approach I haven't thought of yet.

[kabapy]

I found this package

https://github.com/Vinlic/WebVideoCreator/blob/master/README.en-US.md

It works like a charm. It works by rendering the browser content frame by frame. It has some issues wih video in browser, but it is working great and is producing smooth videos on low end hardware