Trying to use onVideoDecode to get an image for ML

[git-rearden]

Hi, I am using a node rstp server into js to get access to ip camera stream. I want to get a "real" image out of an OnVideoDecode handler.

The images I get for png or jpeg are seen as corrupted by gimp and I have not been able to figure out how to adddress this. @phoboslab suggested trying to use currentY Luma place with disableWebAssembly: true.

Suggested using http://player.video.currentY (confused by this url). I am not able to access the currentY array I am able to get the width for example as player.video.codedWidth. Direct assignment/manipulation of player.video.currentY does not seem to be the correct approach. (I do see it in the debugger)

Can anyone suggest either how to get valid png or jpg (sending to a php server for saving to disk, I can see that the preamble iin the files is not correct, have tries to spatch it in code, wo success) and/or the proper way to access currentY. I am thinking I should be able to put the vector into a 2d "image array", but am not having success. Thinking I sb able to get to an opencv js mat from this approach?

Thank you, I am new to jsmpeg. Any thoughts appreciated. I can add code snips to this post - its a bit messy right now with numerous attempts ...

[phoboslab]

Suggested using http://player.video.currentY (confused by this url)

Ha, that was just Twitter confusing player.video.currentY with a URL and adding the http://

Direct assignment/manipulation of player.video.currentY does not seem to be the correct approach.

Why would you want to manipulate currentY? I assumed the goal was to read the decoded pixel data?

So, taking a step back: when using the Canvas2D mode (i.e. disableWebAssembly: true.), you can access the raw YCbCr data in 3 separate arrays, one for each plane: currentY, currentCr and currentCb. In this case player.video.currentY is an Array of width * height elements. Each element contains the "luma" (lightness) value for a single pixel in the video. E.g. player.video.currentY[0] is the top left pixel, player.video.currentY[1] the second one etc.

I am thinking I should be able to put the vector into a 2d "image array", but am not having success.

I don't know anything about OpenCV, but a google search points to cv.matFromArray(). Note that currentY is a flat (one-dimensional) array - which seems to be exactly what matFromArray() expects.

Can anyone suggest either how to get valid png or jpg

So, you don't want the raw pixel data after all, but a complete PNG or JPEG image? Try HTMLCanvasElement.toDataURL(). In JSMpeg you can access the Canvas element at player.renderer.canvas. This works with both, the 2d and WebGL renderers. E.g.:

let base64EncodedImageData = player.renderer.canvas.toDataURL('image/png');

[git-rearden]

First, thank you very much for responding.

Here is a better description of what I am trying to do.

I have rtsp camera sources that I wish to get into js so that I can run tensorflow.js classification models.

I have some working test code that takes an image in png or jpg format and runs a classifier on the image.

I have started with trying to capture an image from jsmpeg to see if I can feed it to the classifiers.

Ultimately, the solution I seek is to process the stream. All the examples I can find use a web cam rather than an ip camera.

HOWEVER, with your suggestion:

let base64EncodedImageData = player.renderer.canvas.toDataURL('image/png');

image.src = base64EncodedImageData;

and "pruning/converting" php code below (called using $ajax on a website for this purpose, 'data' = image.src), I can now make a png that the classifier groks (need to integrate the classifier with the jsmpeg code and process "pruned" frames no need to save to disk per se).

Still in hacked mode needs work to have a usable solution - which btw, is to use cheap eufy cameras in my home (impressed with the cameras and "their" motion and object detection capabilities) to detect people inside "intruders".

Thanks!

<?php

$file_name = $_POST['file_name']; $img = $_POST['data'];

// https://iqcode.com/code/php/php-base64-encoded-image-to-png $img = str_replace('data:image/png;base64,', '', $img); $img = str_replace(' ', '+', $img); $data = base64_decode($img);

echo "from php file_name: ".$file_name."\n"; $fp = fopen($file_name, "w");

fwrite($fp, $data); fclose($fp);