MediaPipe shouldn't resize the external canvas

[tzahola]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

web

MediaPipe Tasks SDK version

0.10.12

Task name (e.g. Image classification, Gesture recognition etc.)

vision

Programming Language and version (e.g. C++, Python, Java)

javascript

Describe the actual behavior

MediaPipe resizes the external canvas to match the size of the input image/video

Describe the expected behaviour

MediaPipe shouldn't resize the external canvas

Standalone code/steps you may have used to try to get what you need

The JS API for mediapipe/tasks-vision lets us pass in an external HTMLCanvasElement via VisionTaskOptions:

/** The options for configuring a MediaPipe vision task. */
declare interface VisionTaskOptions extends TaskRunnerOptions {
    /**
     * The canvas element to bind textures to. This has to be set for GPU
     * processing. The task will initialize a WebGL context and throw an error if
     * this fails (e.g. if you have already initialized a different type of
     * context).
     */
    canvas?: HTMLCanvasElement | OffscreenCanvas;
    /**
     * The running mode of the task. Default to the image mode.
     * Vision tasks have two running modes:
     * 1) The image mode for processing single image inputs.
     * 2) The video mode for processing decoded frames of a video.
     */
    runningMode?: RunningMode;
}

This lets MediaPipe use the WebGL context of an existing canvas, e.g. confidence mask textures will be allocated within this context. Without this option, MediaPipe will create its own internal canvas and WebGL context, which means that the consumer cannot use the resulting textures without copying.

However, when the externally-provided HTMLCanvasElement's dimensions don't match the dimensions of the input image/video provided to e.g. segment() or segmentForVideo(), MediaPipe changes the canvas's dimensions to that of the input. This makes the canvas option pretty much useless, and makes it impossible to use MediaPipe with a single global WebGL context.

Other info / Complete Logs

No response

[schmidt-sebastian]

When you pass us the Canvas, we essentially treat it as our own. You can then still use this canvas to obtain any textures that are bound to it, but you are correct that you should not use it as a visible element in your webpage. This is by design and we likely cannot change this easily.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[tzahola]

You can then still use this canvas to obtain any textures that are bound to it

But then these textures are useless. They belong to the canvas's WebGL context, which now belongs to MediaPipe. copyTexImage2d doesn't work between textures of different contexts either, so there's no way go use the textures or even copy their contents.

Given these constraints, I fail to see how the texture-based API can be used at all. Can you show a working example using the texture-based API in WebGL?

[tzahola]

@schmidt-sebastian could you address the previous comment please?