imageData or imageBitmap as input

[ogewan]

I am trying to figure out how to supply imageData or imageBitmap rather than an HTML Image Element with graphical output.

[ogewan]

thanks to this: https://observablehq.com/@fil/image-to-gpu, I managed to get imageData to work.

[ogewan]

Supporting imageData and imageBitmap as input data seems worthwhile imo, especially since HTML ImageElement requires a DOM and makes an http request every time you change the src

[robertleeplummerjr]

I think you are right.

[WesselKroos]

@robertleeplummerjr Are there (after 3 years) still plans to support an ImageBitmap as input? I'm trying to offload gpu.js to a WebWorker in the browser. But...

...these objects throw an "Could not find a KernelValue for Unknown" exception:

• OffscreenCanvas
• ImageBitmap

...these objects work but are not available in a WebWorker:

• HTMLVideoElement
• HTMLImageElement
• HTMLCanvasElement

...this object works but requires an expensive getImageData() call on an OffscreenCanvas, HTMLVideoElement, HTMLImageElement or HTMLCanvasElement to retrieve:

• ImageData

My current algorithm takes 50ms:

const offscreenCanvas = new OffscreenCanvas();
const offscreenCanvasContext= offscreenCanvas.getContext('2d');
offscreenCanvasContext.drawImage(video);
const imageBitmap = offscreenCanvasContext.transferToImageBitmap();
webWorker.postMessage(imageBitmap , [imageBitmap ])

webWorker.onMessage = (imageBitmap ) => {
  const offscreenCanvas = new OffscreenCanvas();
  const offscreenCanvasContext = offscreenCanvas.getContext('2d');
  offscreenCanvasContext.drawImage(imageBitmap);
  const imageData = offscreenCanvasContext.getImageData() // Takes 40ms
  const output = gpuJsKernel(imageData.data) // Takes 10ms
}

Which gives me the correct output but requires me to first convert the OffscreenCanvas to an ImageData object. It would be way faster if I could parse an OffscreenCanvas or (even better) an ImageBitmap as argument to the kernel.

The current workflow takes 50ms which gives me a video with a framerate of 20fps. Being able to directly use an OffscreenCanvas or ImageBitmap would reduce this to 10ms (100fps). And when I move the workflow out of the WebWorker and directly pass the HTMLVideoElement as argument to the kernel it takes 10ms but also blocks the main thread.

[robertleeplummerjr]

Are there (after 3 years) still plans to support an ImageBitmap as input?

I'm not currently working on it, but it sounds like an interesting task. A lot of the guts of reading special use cases was split into separate classes to cut down on bugs and improve testability. Can you provide an example where you do this within a webgl?

[WesselKroos]

Can you provide an example where you do this within a webgl?

What do you mean by "within a webgl"?

Btw, I was able to make an OffscreenCanvas and ImageBitmap as input work on the main thread by mapping it to the existing HTMLImage type in the gpu-browser.js file. Just by simply adding the mapping to the switch case that decides the argumentTypes.

I'm going to check if this also works in a WebWorker. Can I just create a pullrequest if this works or are there more parts in the code to test before it could be stable?

Edit: ImageBitmap does also work when it is mapped like an HTMLImageElement. (To the HTMLImage, Array(4), Sampler2D, WebGLKernelValueHTMLImage and WebGLKernelValueDynamicHTMLImage types)

[robertleeplummerjr]

GPU.js is a wrapper for webgl 1 and 2 and cpu operations. If you'd like to see what is actually happening (sort of) behind the black box is create a kernel, then run kernel.toString(...arguments) and you'll get all the operations that happened within the kernel.

Now, BE WARNED, variables loose most of their meaning because they are automatically assigned an index as they are used. So things can be very difficult to understand.

Now I mention this because sometimes it makes for an easy way to jack in new ideas or one offs like using a OffscreenCanvas as a means of pushing data around or reading from it.

I'm currently working to see if we can pull values from an OffscreenCanvas, we'll see something later this week.

[robertleeplummerjr]

OffscreenCanvas was added as a readable argument in https://www.npmjs.com/package/gpu.js/v/2.12.0

See unit test for usage: https://github.com/gpujs/gpu.js/blob/e9d3b6696b44080e2336ed29abfa4330405b55a2/test/features/read-offscreen-canvas.js#L15

[WesselKroos]

Oh nice! Good to know that it is available in the new version.

I've been trying to get into webgl yesterday because I noticed that the reason why GPU.js is faster for me is only because the readPixels() method is faster than getImageData(). But trying to render a canvas, image or video to a canvas with WebGL seems to be way harder than I initially thought.

Do you know a good starting point for me to try and learn WebGL? Because GPU.js seems to be overkill for what I want to do: Draw an Image/Video/Canvas to a canvas and get the pixel information via readPixels().

I'm currently tempted to use GPU.js because it is so easy to understand. But I'm only returning the pixel information in my kernel.

[robertleeplummerjr]

Ok, now ImageBitmap is supported in https://www.npmjs.com/package/gpu.js/v/2.14.0.

Version numbers got wonky, my bad.

See unit test for usage: https://github.com/gpujs/gpu.js/blob/2ff65b1ad5766cb7f8c80bd9038739482d3a7a88/test/features/read-image-bitmap.js#L12

[robertleeplummerjr]

Do you know a good starting point for me to try and learn WebGL?

To speak frankly: I cannot stand WebGL 1 or 2. It is a nightmare of duck-tape, hacks, and bugs. If I can, even for light stuff because of understanding, I use GPU.js. If you find it isn't doing what you like, help me to make GPU.js behave, or get involved and learn how to add new features yourself.

There are other alternatives, but it involves trying to work out how webgl works in some way or another. GPU.js makes for the thinnest possible implementation to simply get stuff done. GPU.js also will recompile internally when you switch values from either an array output (both single and unsigned) to a texture, and back. And it also assists with things like how you're packing values together.

If you don't need the more advanced features, simply don't use them.

[robertleeplummerjr]

Added ImageData handling in https://www.npmjs.com/package/gpu.js/v/2.15.0

See unit test for usage: https://github.com/gpujs/gpu.js/blob/c209412d722251a81e5c4af8871fa24e1ae6f631/test/features/read-image-data.js#L11

[robertleeplummerjr]

TL:DR; To be clear ImageData, ImageBitmap, and OffscreenCanvas are all now supported as a 2 dimensional lookup, which gives you a pixel, which is an Array(4).