Export high quality video?

[kingpalethe]

Thanks for this great library and to the citizens of the UK for diligently paying their TV License to the BBC.

This closed issue touches on the question of exporting video from VideoContext: https://github.com/bbc/VideoContext/issues/76

This does seem like something there could be significant demand for.

So far I've been able to export jerky, low frame rate video using WebMWriter: https://github.com/thenickdude/webm-writer-js

I've done it successfully like this:

import VideoContext from "videocontext"
import WebMWriter from "webm-writer"
import { exportLocation } from "../utils/paths";

       const canvas = this.refs.canvas
        const ctx = new VideoContext(canvas);
        const videoNode1 = ctx.video(path.join(__static,
            '/landroid_example-I8JoRYblU.mp4'), 0, 2, { muted: true, loop: true });
        videoNode1.start(0);
        videoNode1.stop(5);
        var videoNode2 = ctx.video(path.join(__static, '/soccert-transparent.webm'), 0, 2, { muted: true, loop: true });
        videoNode2.start(2);
        videoNode2.stop(5);
        var videoNode3 = ctx.video(path.join(__static, '/200-puuWV_57.webm'), 0, 2, { muted: true, loop: true });
        videoNode3.start(3);
        videoNode3.stop(5);
        var combineEffect = ctx.compositor(combineDescription);
        videoNode1.connect(combineEffect);
        videoNode2.connect(combineEffect);
        videoNode3.connect(combineEffect);
        combineEffect.connect(ctx.destination);
        ctx.registerCallback("update", function () {
            console.log("new frame");
            videoWriter.addFrame(canvas);
        });
        const exportPath = exportLocation("testExport.webm")
        console.log(exportPath)
        ctx.registerCallback("ended", function () {
            console.log("Playback ended");
            videoWriter.complete().then(function (webMBlob) {
                 var reader = new FileReader()
                reader.onload = function () {
                    var buffer = new Buffer(reader.result)
                    fs.writeFile(exportPath, buffer, {}, (err, res) => {
                        if (err) {
                            console.error(err)
                            return
                        }
                        console.log('video saved')
                    })
                }
                reader.readAsArrayBuffer(webMBlob)
            });
        });
        var videoWriter = new WebMWriter({
            quality: 0.95,    // WebM image quality from 0.0 (worst) to 1.0 (best)
            fileWriter: null, // FileWriter in order to stream to a file instead of buffering to memory (optional)
            fd: null,         // Node.js file handle to write to instead of buffering to memory (optional)
            // You must supply one of:
            frameDuration: null, // Duration of frames in milliseconds
            frameRate: 24,     // Number of frames per second
        });

        ctx.play();

I'm doing this within an Electron application, so I have access to all the node file system stuff.

So as a proof of concept, this is encouraging. My next challenge is to get a high frame rate, high quality export.

Definitely the export is going to have to be "offline", or, put another way, it's not going to be "realtime."

In the closed issue referenced above, this library is mentioned:

https://github.com/spite/ccapture.js/

It doesn't work yet as a NPM package, see this issue:

https://github.com/spite/ccapture.js/issues/78

And I've been unable to even test it due to this issue:

https://github.com/spite/ccapture.js/issues/87

...But, given access to the Node APIs, I am wondering if this is even the right approach?

There are probably lots of ways to go about this, but I am wondering if anyone here has found the way to make this work reliabily and at full frame rate?

[kingpalethe]

I've put up a bounty for anyone who can make a proof of concept showing high quality video export https://www.bountysource.com/issues/66257768-export-high-quality-video Also -- exporting individual PNGs or images of any kind would be fine also, as those could be easily stitched together with ffmpeg or etc....

[iakashpaul]

I had done a similar capture using Puppeteer for recording videos with css overlays. High quality & no lag(ghosting?) as well. Individual frames are saved & then combined with FFMPEG. Done with Chrome & not Chromium(which is the default in Puppeteer), interested in a demo?

[kingpalethe]

Sure, I have been thinking of puppeteer as the solution to this, but have not tried it yet.

[kingpalethe]

The strategy I am trying currently is to not attempt to use ctx.play();, because that seems to be geared to actually playing the video in real time.

Instead I'm doing this:

        function runIteration(fn, numTimes, delay) {
            var cnt = 0;
            function next() {
                // call the callback and stop iterating if it returns false
                if (fn(cnt) === false) return;
                ++cnt;
                // if not finished with desired number of iterations,
                // schedule the next iteration
                if (cnt < numTimes) {
                    setTimeout(next, delay);
                }
            }
            // start first iteration
            next();

        }
        runIteration(function (i) {
            const timeToSeekTo = i / 24
            console.log(timeToSeekTo)
            ctx.currentTime = timeToSeekTo;
            const url = canvas.toDataURL('image/png');
            const base64Data = url.replace(/^data:image\/png;base64,/, "");
            fs.writeFile(exportLocation(i + '.png'), base64Data, 'base64', function (err) {
                console.log(err);
            });

        }, 240, 500);

So the video is never actually "played"... instead, once every 500ms, the timeline is advanced by 1/24th of a second, and a PNG is made from the canvas.

This outputs a bunch of PNGs, and I can put them together with ffmpeg like this: ffmpeg -framerate 24 -i %d.png output.mp4

I see at least three major problems with my approach however:

1. I've chosen 500ms as a magic number. This is unlikely to be portable to other computer... who knows how long the process of advancing the frame and capturing the PNG should actually take? I really need a reliable and synchronous method of doing this instead. I think this is what CCapture.js is designed to remediate, but it takes pretty drastic measures -- redefining RequestAnimationFrame and etc.

2. In order to make this work, I had to use ffmpeg on all my source video files to force them them to 24fps. Without that, the output is all messed up.

3. For some reason when I do this, I lose the "loop" functionality that I had, here:

var videoNode2 = ctx.video(path.join(__static, 'soccert24.webm'), 0, 2, { muted: true, loop: true });

... the videos are no longer looping. I would guess this has something to do with not actually using the play() method.

[kingpalethe]

I think to do this right, I need to somehow know when the canvas has fully updated, like when this has actually happened: ctx.currentTime = timeToSeekTo; .... so somehow this would have to be put into a Promise, so I knew it was then safe to write the PNG file for the frame?

[kingpalethe]

This is interestesting: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitforselectorselector-options

It looks like puppeteer can wait for something to appear on the page.

[kingpalethe]

After some more tests, currently the best thing I can find is just to wait 500 or 1000ms after each ctx.currentTime, and then save the PNG. To do this right, I think what would be needed is a reliable callback that VideoContext would fire after it had painted the result of ctx.currentTime onto the screen. This function:

registerCallback('update', func)

detailed here: http://bbc.github.io/VideoContext/api/module-VideoContext.html

.... doesn't seem to be when I am looking for. I think I'm confused what it actually does, as far as I can tell it does not fire after each call to ctx.currentTime.

[Unsigno]

Hi @kingpalethe, i fixed CCapture to work with npm. And i can help to fix this if isn't enought with CCapture.

It would be good if you can give feedback on the issue, to keep looking for a solution in case there is an error.

[sdobz]

@kingpalethe I have a solution implemented over here: https://github.com/sdobz/canvas-capture https://youtu.be/5JvAKGEG4Vk

It's implemented as a firefox addon, but the important code is not too hard to follow, see the startCapture and finishCapture methods https://github.com/sdobz/canvas-capture/blob/7fbe4cb045a12ea0cb8b12872cfed40111862c0f/capture.js#L46

[germain-gg]

This is quite a tedious task to achieve.

Using MediaRecorder sounds like a great approach. To me there are a few problems that remain unsolved with that way of doing things:

• What happens if there is buffering
• What happens if the browser is not fast enough to render (do you drop frames?)
• It does not capture audio

Regarding the audio, I have opened a PR #123 to add support for the Web Audio API in VideoContext. This means that it would possible to get an audio stream of the output of VC.

Since MediaRecorder returns a MediaStream there could be a possibility to create a MediaStreamTrack and add it to the exported video. But I can see many scenarios where lipsync would be problematic. The synchronisation of medias would be quite flaky.

Browsers can only render at a max frame rate of 60fps. Which means that processing could take as long as the video lasts depending on the sources frame rate.

Video Context does not provide any ways to know when a frame has been rendered, however browsers do. Every time that something is drawn onto the canvas, the browser internally trigger a "paint" operation. Some headless browsers let you hook in and execute code every time a paint happens. I have not found how to do that with puppeteer just yet.

[kingpalethe]

For my purpose, audio isn't important.

But as you point out, this is the major problem:

Video Context does not provide any ways to know when a frame has been rendered

I'm actually using this in an Electron app, so dealing with Chrome specifically, but as far as I know Chrome does not provide an API to know for sure if a canvas has rendered, which suggests only a pretty bad solution which is just "waiting" 1000ms or so after each ctx.currentTime event, then saving the canvas as PNG. That will almost certainly be unreliable.

So currently I'm trying to find a totally different solution -- to try to run various parts of VideoContext... especially EffectNode -- offline, on the command line. Do it separately for each layer of video, outputting transparent WEBM for each video layer, and then combine it all later with FFMPEG.

To do this I need to find a way to "offline process" GLSL effects on the command line, maybe using FFMPEG or a project like this: https://github.com/glslify/glslify

[Unsigno]

Hi @kingpalethe, VideoContext and CCapture are designed to work on the client side ( browser ), so i don't think they're the right modules for a CLI tool. Anyway with these tools you should be able to export your video from the browser.

I am interested in working on the issue, to export from the browser, if you want.

I was trying to solve the CCapture issue with NPM before, to use it on the solution. But the owner is out of the map and i can't close it. I would appreciate it if we solve this before so i can collect those bounty. I have already sent a PR, just need some feedback if there is some bug or that you close the issue, so i will be able to claim.

[Unsigno]

I was taking a look at the code and doing some test and i have something working updating currentTime on a paused video.

VideoContext don't provide a way to know when a frame is rendered, but will be in the next update once all source nodes are ready. It works syncronously, polling the sources status on every update call, then with a custom update loop we get full control.

Every time you update the currentTime (or on the initialization) in the next update it loads the sources, next updates will check the souces status until all are ready, then in the next update the frame is rendered. It sounds a bit messy and my english is poor, but i can explain more the code if you need it.

Play/Pause functions don't will work on this example to keep it simple and explanatory, there could be some flags or something to keep all working.

<!DOCTYPE html>
<html>
<head>
    <title>VideoContext - Export Example</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <script type="text/javascript" src="./webm-writer-0.2.0.js"></script>
    <!-- https://github.com/thenickdude/webm-writer-js -->
    <script type="text/javascript" src="./download.min.js"></script>
    <!-- https://github.com/rndme/download -->
    <script type="text/javascript" src="./videocontext.js"></script>
</head>
<body>
    <script type="text/javascript">

        window.onload = function(){

            // setup the input framerate manually
            var inputFrameRate = 29;
            // calculate the frame duration
            var frameDuration = 1 / inputFrameRate;

            var canvas = document.getElementById("canvas");
            var exportButton = document.getElementById("export-button");

            // initialize WebMWriter
            var videoWriter = new WebMWriter({
                quality: 1,
                frameRate: inputFrameRate
            });

            // initialize VideoContext with custom loop
            var videoContext = new VideoContext(canvas, undefined, {"manualUpdate" : true});

            /* setup you composition */
            var videoNode1 = videoContext.video("./vinput1.mp4"); // 9 seg
            videoNode1.startAt(0);
            videoNode1.stopAt(9);

            var videoNode2 = videoContext.video("./vinput2.mp4"); // 8 seg
            videoNode2.startAt(4);
            videoNode2.stopAt(10);

            var crossFade = videoContext.transition(VideoContext.DEFINITIONS.CROSSFADE);

            videoNode1.connect(crossFade);
            videoNode2.connect(crossFade);

            crossFade.connect(videoContext.destination);
            crossFade.transition(4,8,0.0,1.0);
            /* end composition */

            // check if all required source nodes are ready
            function nodesReady(){
                for (var i in videoContext._sourceNodes) {
                    if (!videoContext._sourceNodes[i]._ready && videoContext._sourceNodes[i].state < 4) return false;
                }
                return true;
            }

            exportButton.addEventListener("click", function(evt){

                // almost all done by VideoContext is managed on update calls
                // since we manage the timeline using "currentTime" we can update with delta 0 just to keep working the process
                function update(){

                    // when all required sources are ready (so we "catch" the render)
                    if (nodesReady()) {

                        // this update call will render
                        videoContext.update(0)
                        // store the frame some how
                        videoWriter.addFrame(videoContext.element)
                        // seek the next frame
                        videoContext.currentTime += frameDuration
                    }

                    // on load or when we seek the frame it loads the sources, otherwise poll for source status
                    videoContext.update(0)

                    if (videoContext.currentTime < videoContext.duration){ // this way skips the last frame
                        // loop until the end (provably its better using a setTimeout)
                        requestAnimationFrame(update);
                    }else{
                        // then store the result
                        videoWriter.complete().then(function(blob){
                            download( blob, 'testvid.webm', 'video/webm' );
                        });
                    }
                };

                // start the loop
                update();

            }, false);

        }
    </script>

    <div>
        <canvas id="canvas" width="1280", height="720"></canvas>
        <button id="export-button">Export</button>       
    </div>

</body>
</html>

[kingpalethe]

@Unsigno Thanks. I am wondering if this will work. One of the developers here, @gsouquet, wrote on this issue:

Video Context does not provide any ways to know when a frame has been rendered

...... and you seem to be suggesting that you have found a stable way to know when a frame has been rendered.... you seem to trying to do so with

 function nodesReady(){
                for (var i in videoContext._sourceNodes) {
                    if (!videoContext._sourceNodes[i]._ready && videoContext._sourceNodes[i].state < 4) return false;
                }
                return true;
            }

I am suspicious that this may not reliable, because if it were this straightforward, surely @gsouquet would have pointed this out.

But I will definitely be trying this soon and will post back here with my findings when I do.

[Unsigno]

Hi @kingpalethe you have already tried my solution ?

The code you point out indicates that the processing nodes will be applied (calling it before a update), this is not reliable playing the videoContext, since working with deltaTime can skip frames. But with paused videoContext and setting currentTime for a specific frame, yes, it is a stable way to know when the frame is rendered.

It seems straightforward because it is coded, commented and explained. but as you know this does not appear in the documentation, so I had to reverse the code and do some tests to give you all done.

I think it fits your need, but I will be happy to hear your feedback or developer comments to solve the issue.

[Unsigno]

@kingpalethe i think it fits you "runIteration" concept, are you having some issue ??

[kingpalethe]

So it seems the solution to this issue is a custom build of ffmpeg that can acccept GLSL code and use that to render the video on the command line. It's a bit complicated.

[MysteryPancake]

It may still be possible to use MediaRecorder:

Using MediaRecorder sounds like a great approach. To me there are a few problems that remain unsolved with that way of doing things:

What happens if there is buffering?

This could be overcome with requestFrame. "Applications that need to carefully control the timing of rendering and frame capture can use requestFrame() to directly specify when it's time to capture a frame." If requestFrame could be hooked into each canvas update, buffering could be avoided (in theory).

It does not capture audio.

Audio can be captured via MediaStream.addTrack(). It may be possible to render the audio and video separately (perhaps using an OfflineAudioContext), then connect them together via addTrack to keep the timing synchronized.