YUV to RGB Pixel Format

[ianrothmann]

Hi there,

Thanks for a great library. I'm reading a vp8 mkv video file and need to process the frames with TensorFlow JS. The frame data is given in YUV format that I need to get into RGB. I've tried some formulas but something is just not right. Is there an easy way to convert it that anyone knows about?

  let demuxer = await beamcoder.demuxer('/Users/ianrothmann/Downloads/RT76057bde97adbd0537bcdb89037c9034.mkv'); // Create a demuxer for a file
  let decoder = beamcoder.decoder({ name: 'vp8' }); // Codec asserted. Can pass in demuxer.
  let packet = {};

  for ( let x = 0 ; x < 1000 && packet != null ; x++ ) {
    packet = await demuxer.read(); // Read next frame. Note: returns null for EOF
    if (packet && packet.stream_index === 0) { // Check demuxer to find index of video stream
      let frames = await decoder.decode(packet);

      for(let f of frames.frames){
        const frameData=[];
        const width=f.width;
        const height=f.height;
        let [ y, u, v ] = f.data;
        //I need to convert here

            }
          }
      }
    }
  }
  let frames = await decoder.flush();

Much appreciated!

[scriptorian]

Hi, Thanks for your interest in beamcoder.

My understanding is that vp8 is exclusively an 8-bit 4:2:0 format and it is likely that the FFmpeg vp8 decoder will deliver data as 3 separate buffers for Y, U and V with the U & V buffers each being 1/4 the size of the Y buffer. For the conversion you could look in the Streampunk/nodencl project where you will find a colourMaths.js file in the scratch folder with a function that you could use to generate a set of matrix coefficients and a matrixMultiply function that would do the right thing on YUV source data if you are not too worried about the knotty details of white points and gamma functions.

However I think an easier way to progress this could be to use a bit more of beamcoder to do the conversion for you. Using the streaming helpers that would be something like the below, with scale, spec, colorspace etc values set as you require:

async function run() {
  const urls = [ 'file:/Users/ianrothmann/Downloads/RT76057bde97adbd0537bcdb89037c9034.mkv' ];
  const spec = { start: 0, end: 10 };

  const params = {
    video: [
      {
        sources: [
          { url: urls[0], ms: spec, streamIndex: 0 }
        ],
        filterSpec: '[in0:v] scale=1280:720, colorspace=all=bt709 [out0:v]',
        streams: [
          { name: 'rawvideo', time_base: [1, 90000],
            codecpar: {
              width: 1280, height: 720, format: 'rgb24', color_space: 'bt709',
              sample_aspect_ratio: [1, 1]
            }
          }
        ]
      }
    ],
    out: {
      formatName: 'rawvideo',
      url: 'file:temp.raw'
    }
  };

  await beamcoder.makeSources(params);
  const beamStreams = await beamcoder.makeStreams(params);

  await beamStreams.run();
}

console.log('Running');
let start = Date.now();
run()
  .then(() => console.log(`Finished ${Date.now() - start}ms`))
  .catch(console.error);

I hope this helps!

[Watunder]

sample code

const demuxer = await beamcoder.demuxer('assets/video.mp4');
const decoder = beamcoder.decoder({ demuxer: demuxer, stream_index: 0 });
const filterer = await beamcoder.filterer({
    filterType: 'video',
    inputParams: [
        {
            width: decoder.width,
            height: decoder.height,
            pixelFormat: decoder.pix_fmt,
            timeBase: [demuxer.duration, 1],
            pixelAspect: decoder.sample_aspect_ratio,
        }
    ],
    outputParams: [
        {
            pixelFormat: 'rgb24'
        }
    ],
    filterSpec: `scale=${width}:${height}`
});

let packet = beamcoder.packet();
let decResult = null;
let filtResults = null;

for (let step = 0; packet != null; step++) {
    packet = await demuxer.read();
    if (packet && packet.stream_index === 0) {
        decResult = await decoder.decode(packet);
        filtResults = await filterer.filter(decResult.frames);
        if (filtResults[0].frames.length !== 0) {
            //
        }
    }
}

decResult = await decoder.flush();
filtResults = await filterer.filter(decResult.frames);

for (let step = 0; step < filtResults[0].frames.length; step++) {
    //
}