Add mask.getAsImageBitmap() method to vision tools like image segmentation

[kirill-konshin]

MediaPipe Solution (you are using)

image_segmenter

Programming language

No response

Are you willing to contribute it

None

Describe the feature and the current behaviour/state

Current solution returns canvas/texture/intarray/floatarray. Previous MediaPipe solution returned bitmap which was very useful.

Will this change the current API? How?

Create a method like result.confidenceMasks[0].getAsImageBitmap()

Who will benefit with this feature?

Users of BreakoutBox or other stream processing methods where you get ImageBitmap in and ImageBitmap out. intarray/floatarray are CPU intensive, canvas return does not have any image, texture is bound to context of MP and is not useful without extra efforts.

Please specify the use cases for this feature

Even in the example where masks are used for compositing the imagebitmap will produce much more optimized results rather than imagedata since imagebitmap is GPU entity.

Any Other info

No response

[kuaashish]

Hello @lu-wang-g, @schmidt-sebastian,

Could you please have a look into this feature request. Thank you

[schmidt-sebastian]

Thanks for this feature request. This is already on our list of features and we will take a look. We support for ImageBitmap for MPImage, but not for MPMask yet.

[kirill-konshin]

Yes, MPImage supports bitmaps as input.

Minimal example where bitmaps in MPMask output can be useful:

try {

    let bitmapsAndFramesToCleanup;
    const cleanBitmapsAndFrames = () => {
        bitmapsAndFramesToCleanup?.forEach(f => f?.close?.());
        bitmapsAndFramesToCleanup = [];
    };

    const imageSegmenter = await ImageSegmenter.createFromOptions(...);

    const processor = new MediaStreamTrackProcessor({ track: cameraTrack });
    const generator = new MediaStreamTrackGenerator({ kind: 'video' });
    const reader = processor.readable.getReader();
    const writer = generator.writable.getWriter();

    while (cameraTrack.readyState === 'live') {
        const { done, value: frame } = await reader.read();

        if (!frame || done) break;

        bitmapsAndFramesToCleanup = [frame];

        const cameraBitmap = await createImageBitmap(frame); // in actual app this is being sent to Web Worker

        bitmapsAndFramesToCleanup.push(cameraBitmap);

        // in real app this is happening in Web Worker
        const result = await new Promise(res => imageSegmenter.segmentForVideo(cameraBitmap, performance.now(), res));

        const newFrame = new VideoFrame(result.confidenceMasks[0].getAsImageBitmap(), { timestamp: frame.timestamp });
        //                                                        ^^^^^^^^^^^^^^^^
        //                                                        NEW METHOD HERE

        await writer.write(newFrame);

        bitmapsAndFramesToCleanup.push(result, newFrame);

        cleanBitmapsAndFrames();
    }
} catch (e) {
    console.error('MAIN: Error in loop', id, e);
} finally {
    await Promise.all([reader?.cancel(), writer?.close(), generator?.stop(), cleanBitmapsAndFrames()]);
}

[Singulariteehee]

Creating an ImageBitmap from a texture is not very complicated. A helper function that does this would be convenient but not very necessary. It would be nice if the segmentation options could include an option to say that you would really like the output in the form of an ImageBitmap, so that instead of rendering the final output to a texture which then has to be redrawn to the canvas before being converted into an ImageBitmap, the final output would be rendered directly to the canvas instead of to a texture, so that it can be directly copied or transferred to an ImageBitmap without an additional draw step.