Convert frame to Mat

[jose-guerrerog]

Required Info
Camera Model	D435i
Firmware Version	5.12.3.0
Operating System & Version	Win 10
Platform	PC
SDK Version	2.34.0
Language	nodejs
Segment	Robot

Issue Description

Cannot get image in OpenCV format (cv.Mat). I'm following example on opencv-wrapper. When I'm running the code below I'm getting a black screen.

const rs2 = require('node-librealsense');
const cv = require('opencv4nodejs');

const colorizer = new rs2.Colorizer();  // This will make depth image pretty
const pipeline = new rs2.Pipeline();  // Main work pipeline of RealSense camera
pipeline.start();  // Start camera

const frameset = pipeline.waitForFrames(); 
const color = frameset.colorFrame;  // Get RGB image

const matFromArray = new cv.Mat(color.height, color.width, cv.CV_8UC3 , color.getData());

cv.imshow('pic', matFromArray);

cv.waitKey(0);
rs2.cleanup();

[alex-controlx]

This question is related to an integration between RealSense NodeJS library (when using cameras such as D435i) and opencv4nodejs.

Below is the working code:

const rs2 = require('node-librealsense');
const cv = require('opencv4nodejs');

const pipeline = new rs2.Pipeline();  // Main work pipeline of RealSense camera
pipeline.start();  // Start camera

const frameset = pipeline.waitForFrames();  // Get a set of frames
const colorFrame = frameset.colorFrame;  // Get RGB image

const arrayBuffer = new ArrayBuffer(colorFrame.dataByteLength);
colorFrame.getData(arrayBuffer);

// converting ArrayBuffer to NodeJS Buffer
const nodeJsBuffer = Buffer.from(arrayBuffer);

const mat = new cv.Mat(nodeJsBuffer, colorFrame.height, colorFrame.width, cv.CV_8UC3);
cv.imshow('pic', mat);

cv.waitKey(0);
rs2.cleanup();

The approach in the question from @jose-guerrerog doesn't work due to colorFrame.getData() returns UInt8Array, which isn't typical Array of numbers in NodeJS (Array.isArray(uInt8Array) returns false). When cv.Mat required type is number[]. However cv.Mat constructor throws an error when I do the following:

// didn't work
const uInt8Array = colorFrame.getData();
const mat = new cv.Mat(colorFrame.height, colorFrame.width, cv.CV_8UC3, Array.from(uInt8Array));

I don't exactly remember the error. I'll investigate more and post it later.

[alex-controlx]

I've got the error message when I'm using Array.from(uInt8Array):

Mat::New - number of channels (3) do not match fill vector length 2764800

I guess I need to remap the data as it is 1 dimension array as per below:

const matData = []
for(let i = 0; i < rows; i++) {
  matData.push(data.slice(i * cols, cols))
}
const desc = new cv.Mat(matData, cv.CV_8UC3)

but it still didn't work and I got Mat::New - Mat cols must be of uniform length, at column: 1

[alex-controlx]

Got it working

const frameset = pipeline.waitForFrames();
const colorFrame = frameset.colorFrame;
const uint8Array = colorFrame.getData();

const dataMap: number[][][] = [];
const height = colorFrame.height;
const width = colorFrame.width * 3;
for (let i = 0; i < height; i++) {
    const row: number[][] = [];
    for (let k = 0; k < width; k = k + 3) {
        // RGB => BGR
        row.push([uint8Array[i * width + (k + 2)], uint8Array[i * width + (k + 1)], uint8Array[i * width + k]]);
    }
    dataMap.push(row);
}

const colorMat = new cv.Mat(dataMap, cv.CV_8UC3);

[alex-controlx]

@jose-guerrerog this issue can be closed.