[question]: How to receive raw data(texture) instead of an DOM object from the pipeline?

[kalwalt]

In my testing app, developed with OpenCV C++ and Emscripten, my input data are Uint8Array, I'm passing data from a video stream into a speedy-vision pipeline and then the output ImageBitmap is transformed in a ImageData by s simple convert routine:

// note image.source is ImageBitmap
ctx.drawImage(image.source, 0, 0);
data = ctx.getImageData(0, 0, oWidth, oHeight);

then i pass the ImageData.data in a worker when needed.

Looking in the speedy-vision code you can't get a Uint8Array from a speedy-media-source. Does it possible to pass raw data as Uint8Array from the pipeline instead doing that conversion? Or there is an alternative approach? In this sense i could avoid this conversion.

[alemart]

A Uint8Array is just data. We don't know if it's an image, and if it is, we don't know its dimensions nor its format. On the other hand, an ImageData includes dimensions and pixel data in the RGBA format.

If you inspect the code of the Image Sink at src/core/pipeline/nodes/images/sink.js, you'll see that a texture is first rendered to a canvas, and then an ImageBitmap is created from it. You could make a variant of that sink, so that an ImageData is created instead.

[kalwalt]

Hi @alemart thank you for the answer. 🙂

If you inspect the code of the Image Sink at src/core/pipeline/nodes/images/sink.js, you'll see that a texture is first rendered to a canvas, and then an ImageBitmap is created from it. You could make a variant of that sink, so that an ImageData is created instead

Do you mean that it's needed to modify the source code?

[alemart]

Do you mean that it's needed to modify the source code?

Yes. Instead of converting an ImageBitmap into an ImageData, you would generate an ImageData directly.

[alemart]

SpeedyTextureReader.readPixelsSync (slow) or SpeedyTextureReader.readPixelsAsync (better) will give you pixel data in RGBA format, as an Uint8Array. texture.width and texture.height will give you the dimensions of the image. You can create an ImageData with that, or just output the pixel data directly if it suits you.

[kalwalt]

@alemart thank you for the inputs i will try for sure!

[kalwalt]

I'm testing a new sink in this PR https://github.com/kalwalt/webarkit-testing/pull/24 i will let you know how it goes. :slightly_smiling_face:

[kalwalt]

So i created a new sink node, basically a copy of the original node but the run method return ImageData:

run(gpu)
   {
       const { image, format } = /** @type {SpeedyPipelineMessageWithImage} */ ( this.input().read() );

       return new SpeedyPromise(resolve => {
           const canvas = gpu.renderToCanvas(image);
           const ctx = canvas.getContext('2d');
           ctx.drawImage(image.source, 0, 0);
           ctx.getImageData(0, 0, image.width, image.height).data.then(data => {
               //this._imageData = data;
               this._bitmap = data;
               this._format = format;
               resolve();});
       });
   }

but i got this error:

Uncaught (in promise) a: Illegal argument. Can't connect port in of "n[image]" to port out of "Kn[7d0433e9b9a76]": incompatible types
-> [speedy-vision.js]
    at B.connectTo (http://localhost:63342/webarkit-testing/dist/SpeedyVisionSinkImageData.js:1:189399)
    at Wt.connectTo (http://localhost:63342/webarkit-testing/node_modules/speedy-vision/dist/speedy-vision.min.js:10:252294)
    at loadSpeedyImage (http://localhost:63342/webarkit-testing/examples/init_speedy.js:22:24)

i suppose that because ImageData is not an ImageMedia type, so i think i have to make slightly changes directly inside speedy-vision. i would avoid this but i think i have no choice if i want his feature.

P.E.: the new node is sink-image-data.js

[alemart]

Are you trying to input an ImageData into a pipeline? I might look into adding support.

i would avoid this but i think i have no choice if i want his feature.

hmmm, maybe you could just export() a SpeedyPromise<ImageData> instead, as in return SpeedyPromise.resolve(imageData); ?

[kalwalt]

Are you trying to input an ImageData into a pipeline? I might look into adding support.

i would avoid this but i think i have no choice if i want his feature.

hmmm, maybe you could just export() a SpeedyPromise<ImageData> instead, as in return SpeedyPromise.resolve(imageData); ?

yes that's was the idea, but maybe require more works and more changes in the speedy-vision core. Instead i can try as you suggested.

[kalwalt]

i have tried in the export() but i get again:

SpeedyVisionSinkImageData.js:1 Uncaught (in promise) a: Illegal argument. Can't connect port in of "n[image]" to port out of "Kn[65063ab21018b]": incompatible types
-> [speedy-vision.js]
    at N.connectTo (http://127.0.0.1:5500/dist/SpeedyVisionSinkImageData.js:1:186386)
    at Wt.connectTo (http://127.0.0.1:5500/node_modules/speedy-vision/dist/speedy-vision.min.js:10:252294)
    at loadSpeedyImage (http://127.0.0.1:5500/examples/init_speedy.js:22:24)

code in this commit. and note that i fallback the _run() method to the original code. So i thinking that maybe this is not allowed outside of speedy-vision core and then i will try to create a new class inside speedy-vision (on my fork) as i did here.

[kalwalt]

Working on this branch for the ImageData feature, i will send a PR very soon for this.

POST EDIT: related PR #56