method `toPixels` is not optimized for `webgl` or `webgpu` backends

[vladmandic]

method fromPixels uses optimized code paths depending on the browser and using webgl calls results in 10x faster performance than fallback using canvas.getImageData

however, method toPixels does not have any optimizations and relies strictly on downloading tensor, creating ImageData object from it and then drawing it onto canvas
(implementation is in src/tfjs-core/src/opts/browser.ts:toPixels

for models where result is already in GPU memory, tensor download is by far the most expensive (and unnecessary) operation

for example, look at the following timing values (in ms):

• prepare: 1 //includes call to fromPixels and normalizing inputs to -1..1
• inference: 2 // call to await model.executeAsync
• process: 0 // includes denormalizing and converting data from rgb to rgba
• download: 147 // this is just await tensor.data()
• draw: 2 // actual canvas.drawImage
• total: 153

as you can see this may be an extreme case, but 96% of time is spent on unnecessary download of data from gpu memory

ask is to implement optimized webgl and webgpu path for toPixels method

environment: tfjs 3.19.0 with chrome 103

[qjia7]

@vladmandic Does dataToGPU satisfy your requirement? Currently, both webgl and webgpu support dataToGPU which directly exposes the underlying GPU resources (texture for webgl, buffer for webgpu). You can do any post-processing using the exposed gpu resource. For example, you can custom your own toPixels by rendering the retuned gpu resource to the canvas.

[vladmandic]

@qjia7 yup, aware of dataToGPU, but i'm not a GLSL expert, so it will take me a while to come up with a working code.
just feels natural that toPixel should have optimized path equal to fromPixels

[qjia7]

Personally, I think toPixels's interface async function toPixels(img: Tensor2D|Tensor3D|TensorLike, canvas?: HTMLCanvasElement): Promise<Uint8ClampedArray> is problematic, which requires it must return a Uint8ClampedArray. That means it's unavoidable to download data from gpu to cpu. To optimize toPixels, we need to modify the current interface to avoid return a Uint8ClampedArray, which is a breaking change for this API. So one way is to deprecate this API and provide a new interface for it, which can be easily optimized in webgl/webgpu. The other is to provide a document/example to tell users how to use dataToGPU to implement toPixels's functionality. @vladmandic @pyu10055 How do you think?

[vladmandic]

@qjia7 example is always useful to have (currently there is not much about dataToGPU) and can be used immediately

on the breaking change for toPixels, well, some changes can be good - i see no reason why it should return pixel data to start with. my suggestion would be to keep compatibility with old behavior via env flag instead of introducing new api.

[vladmandic]

[pyu10055]

@vladmandic Please check out the code example on how to use dataToGPU with WebGL.

[vladmandic]

@pyu10055 gpu-pipeline example is 500+ lines of code split into 6 js files (not modules) that are visible to each other only via global namespace - this kind of project structure is a nightmare to follow, opposite of what an example should be

and actual processing method takes gl texture from tensor and returns gl framebuffer that is later used for drawing on canvas, so its not even directly applicable without modifying tons of code.

let just say that my feature request stands - to provide out-of-the-box optimized toPixels method.

[qjia7]

@vladmandic I ever wrote a PR to optimize toPixels. After some internal discussions, dataToGPU is a more powerful interface for users. So I suspended that work. It seems that it's still useful to provide an efficient way for toPixels. One question here is what kind of canvas would you like to draw to? A 2d-canvas or let the backend to decide the canvas context (webgl/webgpu context canvas)? The original toPixels seems directly use it as a 2d-canvas.

[vladmandic]

@qjia7 imo, backend to decide unless canvas already has a context - in which case it should fall back to using whatever is already allocated, but its ok if its unoptimized in that case.

[vladmandic]

got a working example with single typescript module (200 loc, with full strong typing) and with a single exported method: source code: https://github.com/vladmandic/anime/blob/main/src/gldraw.ts

usage:

const data = tensor.dataToGPU({ customTexShape: [canvas.width, canvas.height] }); // get pointer to tensor texture on gpu
drawTexture(canvas, data.texture); // draw texture on canvas
tf.dispose(data.tensorRef); // dispose tensor

[pyu10055]

@vladmandic your snippet capture the essense of the API, thanks. @lina128 Please take a look at the comments, maybe we should add this example to the API doc.

[lina128]

Thank you for the example contribution @vladmandic . Can we reference your example in the API doc?

[vladmandic]

@lina128 sure!

[ygf8]

@vladmandic Thank you for the example Is it necessary to use syncWait to draw again ? Syncwait takes time

[vladmandic]

@ygf8 its not necessary to call syncWait to draw data at all. the issue is that some model execution has items on webgl pipeline even after model.execute finishes, so if you run a model in an immediate loop, it can lock up system eventually.

when you're using await tensor.data or tensor.dataSync, they internally do some syncWaits so you never see that problem. but with this example, there is no data download ever, so every now and then its recommended to perform a syncWait which waits for webgl pipeline to clear up.

[ygf8]

@vladmandic Thank you I understand now. Is there any way to force the webgl cleanup associated to model execution or reset it because the model has already given the result we expected? I'm not really sure if it can speed up the process or maybe use a multi-threaded system to shorten the time between two predictions+drawing

[vladmandic]

Is there any way to force the webgl cleanup associated to model execution or reset it because the model has already given the result we expected

Maybe, but i cannot think of anything better than this syncWait.
But you don't need to call it on every frame, in my experience just every now and then to clean up the webgl pipeline (how often is model and system specific, but every couple of seconds is sufficient in my experience).

And if you don't wait for webgl pipeline cleanup, you'd still see fast inference/draw times reported,
but browser itself starts dropping frames and slows down drastically.

maybe use a multi-threaded system to shorten the time between two predictions+drawing

Chrome only maintains one webgl pipeline regardless of number of threads.

[ygf8]

@vladmandic thank you again , I'm going to explore all this :)

[FabioRomagnolo]

got a working example with single typescript module (200 loc, with full strong typing) and with a single exported method: source code: https://github.com/vladmandic/anime/blob/main/src/gldraw.ts

usage:

const data = tensor.dataToGPU({ customTexShape: [canvas.width, canvas.height] }); // get pointer to tensor texture on gpu
drawTexture(canvas, data.texture); // draw texture on canvas
tf.dispose(data.tensorRef); // dispose tensor

Thanks for your solution. Is there any easier-to-use javascript version of drawTexture() function around? I wish to include it in a simple script html tag.

[vladmandic]

@FabioRomagnolo just compile to JS (which in this case doesn't do anything but strip typedefs from TS source)
for example tsc --target es2018 --module es6 gldraw.ts

[FabioRomagnolo]

@FabioRomagnolo just compile to JS (which in this case doesn't do anything but strip typedefs from TS source) for example tsc --target es2018 --module es6 gldraw.ts

Thanks for you reply! Trying to draw an inference result I got the following error:

gldraw.js:16 WebGL: INVALID_OPERATION: bindTexture: object does not belong to this context

So I tried compiling also glbackend.js in order to use registerWebGLbackend() function on my canvas object, but I can't find your @vladmandic/tfjs/dist/tfjs.esm package.

This is the command I launched to compile the glbackend.ts file:

tsc --target es2018 --module es6 --moduleResolution node glbackend.ts

[vladmandic]

@vladmandic/tfjs is just my build of latest version of tfjs directly from main branch. you can use regular @tensorflow/tfjs

[FabioRomagnolo]

@vladmandic/tfjs is just my build of latest version of tfjs directly from main branch. you can use regular @tensorflow/tfjs

Yes, I sensed that but I get the tf.setWebGLContext is not a function error.

[vladmandic]

that function comes from @tensorflow/tfjs-backend-webgl which i'd assume you're using since this is all about webgl anyhow...

[FabioRomagnolo]

that function comes from @tensorflow/tfjs-backend-webgl which i'd assume you're using since this is all about webgl anyhow...

Placing the import of @tensorflow/tfjs-backend-webgl after the @tensorflow/tfjs one I get some warning about registering twice the WebGL operations, but I don't think this is a problem.

Anyway, I tried to register the custom WebGL backend both for TF environment and my canvas object. Then I drew the result, but I got a weird image like the following.

await registerWebGLbackend(document.createElement('canvas'));
await tf.setBackend('customgl');
...
await registerWebGLbackend(liveOutputCanvas);
...
const data = tfgr.dataToGPU({ customTexShape: [liveOutputCanvas.width, liveOutputCanvas.height] });
drawTexture(liveOutputCanvas, data.texture);
await tf.dispose(data.tensorRef);
await syncWait(tf.backend().getGPGPUContext().gl); 

There must be something that still I'm missing.

[vladmandic]

the shape does not match (width, height, depth) - what is the shape of your tensor?

• auto-resize is flaky, its best if width/height matches canvas width/height
• if depth is rgb (3), you need to convert it to rgba (4)
  (see <https://github.com/vladmandic/anime/blob/78a503975c1b5bb74e2bc123e74da9ceffed8882/src/anime.ts#L43)

also, i may have bugs in non-square tensors (tfjs uses height/width while rest of js uses width/height), not a big deal but i need to test with a non-square model.

[FabioRomagnolo]

the shape does not match (width, height, depth) - what is the shape of your tensor?

• auto-resize is flaky, its best if width/height matches canvas width/height
• if depth is rgb (3), you need to convert it to rgba (4) (see <https://github.com/vladmandic/anime/blob/78a503975c1b5bb74e2bc123e74da9ceffed8882/src/anime.ts#L43)

also, i may have bugs in non-square tensors (tfjs uses height/width while rest of js uses width/height), not a big deal but i need to test with a non-square model.

The RGBA tensor I'm trying to show has the standard TF shape: [height, width, 4]

[vladmandic]

@FabioRomagnolo any chance you can share your model so i can test?

[19521178]

@vladmandic I've tried using your api following anime.ts. I realized that rendering will be successful if tensor is squared, either is like result image of @FabioRomagnolo. You've tested on non-squared model, haven't you? I really need a solution for non-squared tensor but I couldn't solve it by myself.

[FabioRomagnolo]

@vladmandic I've tried using your api following anime.ts. I realized that rendering will be successful if tensor is squared, either is like result image of @FabioRomagnolo. You've tested on non-squared model, haven't you? I really need a solution for non-squared tensor but I couldn't solve it by myself.

Hi! I solved this problem by configuring a custom WebGL 2.0 environment for TensorFlow.js by using the same canvas on which we want to draw.

This is a full explained snippet from my recent demo at https://www.backgroundmatting.it/live_demo and I'm available to answer any question about it :)

1) Prepare WebGL 2.0 custom environment:

/*
REQUIREMENTS:
- TensorFlow.js: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.19.0/dist/tf.min.js"></script>
- TensorFlow.js WebGL backend: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl@3.19.0/dist/tf-backend-webgl.js"></script>
- `gl-util.js`, `gl-class.js`, `gl-shaders.js` modules from https://github.com/tensorflow/tfjs-examples/tree/master/gpu-pipeline
*/
import {getWebGLRenderingContext} from "./webgl/gl-util.js";

const customBackendName = 'custom-webgl2';
const kernels = tf.getKernelsForBackend('webgl');
kernels.forEach(kernelConfig => {
    const newKernelConfig = {...kernelConfig, backendName: customBackendName};
    tf.registerKernel(newKernelConfig);
});
const gl = getWebGLRenderingContext(document.getElementById('resultCanvas'));
tf.registerBackend(customBackendName, () => {
    return new tf.MathBackendWebGL(
        new tf.GPGPUContext(gl));
});
tf.setBackend(customBackendName);
await tf.ready();

2) Draw the tensor as texture. You can import the following function and use it:

/*
REQUIREMENTS:
- TensorFlow.js: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.19.0/dist/tf.min.js"></script>
- TensorFlow.js WebGL backend: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl@3.19.0/dist/tf-backend-webgl.js"></script>
- `gl-util.js`, `gl-class.js`, `gl-shaders.js` modules from https://github.com/tensorflow/tfjs-examples/tree/master/gpu-pipeline
*/
import {createTexture} from "./gl-util.js";
import {RenderTensor} from "./gl-class.js";

export function showTensorInCanvasGL(tensor, height, width, canvasGL){
    /* WARNING! WebGL context must be initialized in the main script.
       Example:
           import {getWebGLRenderingContext} from "../utils/webgl/gl-util.js";
           var canvasGL = getWebGLRenderingContext(resultCanvas);
    */
    let renderTensorGL = new RenderTensor(canvasGL);
    /* WARNING! dataToGPU keeps the tensor on GPU, but it becomes an
                RGBA texture for WebGL, so the tensor should not be RGB.
    */
    let data = tensor.dataToGPU({customTexShape: [height, width]});

    // Drawing texture
    let result = renderTensorGL.process(createTexture(canvasGL, data.texture, width, height));
    canvasGL.bindFramebuffer(canvasGL.DRAW_FRAMEBUFFER, null);
    canvasGL.bindFramebuffer(canvasGL.READ_FRAMEBUFFER, result.framebuffer_);
    canvasGL.blitFramebuffer(0, 0, width, height, 0, height, width, 0,
        canvasGL.COLOR_BUFFER_BIT, canvasGL.LINEAR);
    canvasGL.flush();

    // Cleaning memory
    tensor.dispose();
    data.tensorRef.dispose();
}

[19521178]

@FabioRomagnolo It worked for me. This solution is the best. Thanks for your codes.

[FabioRomagnolo]

@FabioRomagnolo It worked for me. This solution is the best. Thanks for your codes.

You're welcome! :)

[Apj-infinty]

@vladmandic I've tried using your api following anime.ts. I realized that rendering will be successful if tensor is squared, either is like result image of @FabioRomagnolo. You've tested on non-squared model, haven't you? I really need a solution for non-squared tensor but I couldn't solve it by myself.

Hi! I solved this problem by configuring a custom WebGL 2.0 environment for TensorFlow.js by using the same canvas on which we want to draw.

This is a full explained snippet from my recent demo at https://www.backgroundmatting.it/live_demo and I'm available to answer any question about it :)

1) Prepare WebGL 2.0 custom environment:

/*
REQUIREMENTS:
- TensorFlow.js: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.19.0/dist/tf.min.js"></script>
- TensorFlow.js WebGL backend: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl@3.19.0/dist/tf-backend-webgl.js"></script>
- `gl-util.js`, `gl-class.js`, `gl-shaders.js` modules from https://github.com/tensorflow/tfjs-examples/tree/master/gpu-pipeline
*/
import {getWebGLRenderingContext} from "./webgl/gl-util.js";

const customBackendName = 'custom-webgl2';
const kernels = tf.getKernelsForBackend('webgl');
kernels.forEach(kernelConfig => {
    const newKernelConfig = {...kernelConfig, backendName: customBackendName};
    tf.registerKernel(newKernelConfig);
});
const gl = getWebGLRenderingContext(document.getElementById('resultCanvas'));
tf.registerBackend(customBackendName, () => {
    return new tf.MathBackendWebGL(
        new tf.GPGPUContext(gl));
});
tf.setBackend(customBackendName);
await tf.ready();

2) Draw the tensor as texture. You can import the following function and use it:

/*
REQUIREMENTS:
- TensorFlow.js: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.19.0/dist/tf.min.js"></script>
- TensorFlow.js WebGL backend: <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl@3.19.0/dist/tf-backend-webgl.js"></script>
- `gl-util.js`, `gl-class.js`, `gl-shaders.js` modules from https://github.com/tensorflow/tfjs-examples/tree/master/gpu-pipeline
*/
import {createTexture} from "./gl-util.js";
import {RenderTensor} from "./gl-class.js";

export function showTensorInCanvasGL(tensor, height, width, canvasGL){
    /* WARNING! WebGL context must be initialized in the main script.
       Example:
           import {getWebGLRenderingContext} from "../utils/webgl/gl-util.js";
           var canvasGL = getWebGLRenderingContext(resultCanvas);
    */
    let renderTensorGL = new RenderTensor(canvasGL);
    /* WARNING! dataToGPU keeps the tensor on GPU, but it becomes an
                RGBA texture for WebGL, so the tensor should not be RGB.
    */
    let data = tensor.dataToGPU({customTexShape: [height, width]});

    // Drawing texture
    let result = renderTensorGL.process(createTexture(canvasGL, data.texture, width, height));
    canvasGL.bindFramebuffer(canvasGL.DRAW_FRAMEBUFFER, null);
    canvasGL.bindFramebuffer(canvasGL.READ_FRAMEBUFFER, result.framebuffer_);
    canvasGL.blitFramebuffer(0, 0, width, height, 0, height, width, 0,
        canvasGL.COLOR_BUFFER_BIT, canvasGL.LINEAR);
    canvasGL.flush();

    // Cleaning memory
    tensor.dispose();
    data.tensorRef.dispose();
}

Hiii, @FabioRomagnolo , I can't seem to find the function RendorTensor inside anywhere(not in gl-class as you mention nor anywhere inside tfjs-backend-webgl) . @vladmandic I am getting the same output as @FabioRomagnolo mentioned earlier even after giving two separate Backends for output canvas and webGl canvas . Can you guys guide me out here?