'Pipe' raw image data to videoconverter.js

[gfwilliams]

This is an absolutely awesome project - I've been waiting for something like this for ages!

Suppose I had a source of a lot of RGB pixel data coming out of JavaScript on a web page (maybe a Canvas or WebGL context) - how would I go about getting this into videoconverter?

There are examples of what you'd do on the command-line for ffmpeg, like these - but once I'd set up a web worker as in your examples, how could I go about pushing data into stdin?

The idea is I might be rendering something maybe a few minutes long with WebGL, and there would be too much data to store in RAM - so ideally I'd push each frame into the webworker when it was ready.

... or is it not possible as it is - since there's just the ffmpeg_run function which only returns when it's finished?

I guess potentially I could call ffmpeg with a few frames of uncompressed data, and ask it to append them to an existing video?

[gfwilliams]

So... I hacked something up using multiple calls to ffmpeg.

I used the code from the demo page, but tweaked runCommand to take a callback argument. The following just creates a very hacky mandelbrot fractal:

var WIDTH = 160;
var HEIGHT = 120;

sampleVideoData = new Uint8Array(WIDTH*HEIGHT*3*20);

var zoom = 1.0;

function render() {
 for (var f=0;f<20;f++) {
  zoom -= 0.005;
  for (y=0;y<HEIGHT;y++) {
   for (x=0;x<WIDTH;x++) {
    var Xr=0;
    var Xi=0;
    var Cr=(zoom*4.0*x/HEIGHT)-2.0*zoom;
    var Ci=(zoom*4.0*y/HEIGHT)-2.0*zoom;
    var i=0;
    while ((i<5) && ((Xr*Xr+Xi*Xi)<4)) {
      var t=Xr*Xr - Xi*Xi + Cr;
      Xi=2*Xr*Xi+Ci;
      Xr=t;
      i++;
    }
      sampleVideoData[(f*WIDTH*HEIGHT*3)+(x+(y*WIDTH))*3] = (i&1) ? 255 : 0;
   }
  }
 }
}

// ...
    render();
    runCommand("-f rawvideo -vcodec rawvideo -s "+WIDTH+"x"+HEIGHT+" -pix_fmt rgb24 -framerate 10 -i input.raw -an output.webm", [        
      { "name": "input.raw", "data": sampleVideoData }], function(files) {
      var file1 = new Uint8Array(files[0].data);
      render();
      runCommand("-f rawvideo -vcodec rawvideo -s "+WIDTH+"x"+HEIGHT+" -pix_fmt rgb24 -framerate 10 -i input.raw -an output.webm", [        
        { "name": "input.raw", "data": sampleVideoData }], function(files) {
        var file2 = new Uint8Array(files[0].data);
        runCommand('-i a.webm -i b.webm -v debug -strict -2 -filter_complex "[0:v] [1:v] concat=n=2:v=1:a=0 [v]" -map "[v]" -an output.webm', [     
        { "name": "a.webm", "data":file1 }, { "name": "b.webm", "data":file2 }], function(files) {
          console.log("Done");
        });
      });
    });

However the file I get out the end is hugely corrupted. It still shows bits of what it should, but with loads of MPEG artefacts.

To make it work, I had to take the output of one call (which was a normal ArrayBuffer) and wrap it in a Uint8Array in order to feed it back in to the other - could it be some strange kind of UTF8 issue?

[gfwilliams]

Actually that's totally misleading. It turns out that even -f rawvideo -vcodec rawvideo -s 160x120 -pix_fmt rgb24 -framerate 10 -i input.raw -an output.webm produced corrupted video.

However -f rawvideo -vcodec rawvideo -s 160x120 -pix_fmt rgb24 -framerate 10 -i input.raw -an output.gif is fine (but then I can't use concat with it). Any thoughts?

[gfwilliams]

Ok, not sure what was happening there. It just doesn't seem to like compressing to webm the way I was doing it...

The following works, compressing to mp4:

   render();
    var ext = "mp4";
    var encoding = "-c:v libx264";
    runCommand("-f rawvideo -vcodec rawvideo -s "+WIDTH+"x"+HEIGHT+" -pix_fmt rgb24 -framerate 10 -i input.raw -an -vf showinfo "+encoding+" output."+ext, [        
      { "name": "input.raw", "data": sampleVideoData }], function(files) {
      var file1 = new Uint8Array(files[0].data);
      render();
      runCommand("-f rawvideo -vcodec rawvideo -s "+WIDTH+"x"+HEIGHT+" -pix_fmt rgb24 -framerate 10 -i input.raw -an -vf showinfo "+encoding+" output."+ext, [        
        { "name": "input.raw", "data": sampleVideoData }], function(files) {
        var file2 = new Uint8Array(files[0].data);
        runCommand('-i a.'+ext+' -i b.'+ext+' -filter_complex "[0:v] [1:v] concat=n=2:v=1:a=0 [v]" -map "[v]" -an '+encoding+' output.'+ext, [     
        { "name": "a."+ext, "data":file1 }, { "name": "b."+ext, "data":file2 }], function(files) {
          console.log("Done");
        });
      });
    });