search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.82k stars 5.18k forks source link

Mediapipe ImageSegmenter doesn't work in Mozilla and Android browsers #4501

Closed annysmolyan closed 1 year ago

annysmolyan commented 1 year ago

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Linux, Ubuntu,

Mobile device if the issue happens on mobile device

Pixel 6

Browser and version if the issue happens on browser

Mozilla (v 113.0.1), Google Chrome

Programming Language and version

JS

MediaPipe version

No response

Bazel version

No response

Solution

ImageSegmenter

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

always get empty threshold mask result when use vision task with image segmenter on Mobile and Mozilla desktop.

Cheked also with my custom js (using knockout)

this.filesetResolver = await FilesetResolver.forVisionTasks(data.visionWasmUrl);
this.hairSegmenter = await ImageSegmenter.createFromOptions(this.filesetResolver, {
                        baseOptions: {
                            modelAssetPath: [path_here],
                            delegate: "GPU"
                        },
                        outputCategoryMask: false,
                        outputConfidenceMasks: true,
                        runningMode: "VIDEO"
                    });

this.hairSegmenter.segmentForVideo(this.videoElementHTMLObject,startTimeMs,this.drawHair)

//a piece of code from drawHair function:

const mask = result.confidenceMasks[this.RESULT_HAIR_INDEX].getAsFloat32Array();
 for (let i = 0; i < mask.length; ++i) {
  if (mask[i] > 0.5) // ->>>>>  here mask[i] always = 0

even if open demo https://mediapipe-studio.webapps.google.com/demo/image_segmenter on Pixels or Mozilla it doesn't work

Describe the expected behaviour

As expected result I want to get a real hair mask result with not empty threshold

Standalone code/steps you may have used to try to get what you need

this.filesetResolver = await FilesetResolver.forVisionTasks(data.visionWasmUrl);
this.hairSegmenter = await ImageSegmenter.createFromOptions(this.filesetResolver, {
                        baseOptions: {
                            modelAssetPath: [path_here],
                            delegate: "GPU"
                        },
                        outputCategoryMask: false,
                        outputConfidenceMasks: true,
                        runningMode: "VIDEO"
                    });

this.hairSegmenter.segmentForVideo(this.videoElementHTMLObject,startTimeMs,this.drawHair)

//a piece of code from drawHair function:

const mask = result.confidenceMasks[this.RESULT_HAIR_INDEX].getAsFloat32Array();
 for (let i = 0; i < mask.length; ++i) {
  if (mask[i] > 0.5) // ->>>>>  here mask[i] always = 0

Other info / Complete Logs

logs from mozilla:

Content-Security-Policy warnings 2
Some cookies are misusing the recommended “SameSite“ attribute 86
Component analytics tracker already installed for alkaliAccountSelectorTracker 83a8e44533e4943485fa67687487242f07b71693.js:1924:1735
Component analytics tracker already installed for alkaliFeedbackWidgetTracker 83a8e44533e4943485fa67687487242f07b71693.js:1924:1735
Component analytics tracker already installed for alkaliTosTracker 83a8e44533e4943485fa67687487242f07b71693.js:1924:1735
Detected build id: alkali.mediapipestudio_20230510_1517_RC00 as alkali.mediapipestudio_20230510_1517_RC00 5 83a8e44533e4943485fa67687487242f07b71693.js:407:93
Detected build id: alkali.mediapipestudio_20230510_1517_RC00 as alkali.mediapipestudio_20230510_1517_RC00 83a8e44533e4943485fa67687487242f07b71693.js:407:93
Detected server environment: PRODUCTION as GCS prod 83a8e44533e4943485fa67687487242f07b71693.js:406:411
Detected build id: alkali.mediapipestudio_20230510_1517_RC00 as alkali.mediapipestudio_20230510_1517_RC00 83a8e44533e4943485fa67687487242f07b71693.js:407:93
WARNING: Multiple instances of Three.js being imported. ad89869963b07077c4330d0e32a998a2cc5cbd27.js:6:572987
wasm streaming compile failed: TypeError: WebAssembly: Response has unsupported MIME type 'application/octet-stream' expected 'application/wasm' 193cd492c0b2a967a4912923278c1ac367d05d47.js:363:8
falling back to ArrayBuffer instantiation 193cd492c0b2a967a4912923278c1ac367d05d47.js:364:8
I0000 00:00:1685625572.210000       1 gl_context_webgl.cc:112] Successfully created a WebGL context with major version 3 and handle 1 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625572.211000       1 gl_context.cc:359] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)), renderer: Intel(R) HD Graphics 400 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
W0000 00:00:1685625572.215000       1 gl_context.cc:1002] OpenGL error checking is disabled 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph successfully started running. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
INFO: Created TensorFlow Lite XNNPACK delegate for CPU. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625572.221000       1 segmentation_postprocessor_gl.cc:378] NONE activation function chosen on GPU 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph finished closing successfully. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
I0000 00:00:1685625572.244000       1 gl_context_webgl.cc:130] Successfully destroyed WebGL context with handle 1 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625572.246000       1 gl_context_webgl.cc:112] Successfully created a WebGL context with major version 3 and handle 28 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625572.247000       1 gl_context.cc:359] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)), renderer: Intel(R) HD Graphics 400 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph successfully started running. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
Found Custom Op: Convolution2DTransposeBias 2 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.287000       1 segmentation_postprocessor_gl.cc:378] NONE activation function chosen on GPU 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph finished closing successfully. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
I0000 00:00:1685625574.316000       1 gl_context_webgl.cc:130] Successfully destroyed WebGL context with handle 28 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.319000       1 gl_context_webgl.cc:112] Successfully created a WebGL context with major version 3 and handle 423 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.323000       1 gl_context.cc:359] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)), renderer: Intel(R) HD Graphics 400 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph successfully started running. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
Found Custom Op: Convolution2DTransposeBias 2 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.545000       1 segmentation_postprocessor_gl.cc:378] NONE activation function chosen on GPU 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph finished closing successfully. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
I0000 00:00:1685625574.571000       1 gl_context_webgl.cc:130] Successfully destroyed WebGL context with handle 423 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.573000       1 gl_context_webgl.cc:112] Successfully created a WebGL context with major version 3 and handle 818 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.577000       1 gl_context.cc:359] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)), renderer: Intel(R) HD Graphics 400 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
Graph successfully started running. 193cd492c0b2a967a4912923278c1ac367d05d47.js:1070:8
WebGL warning: drawArraysInstanced: Tex image TEXTURE_2D level 0 is incurring lazy initialization. 2
Found Custom Op: Convolution2DTransposeBias 2 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
I0000 00:00:1685625574.838000       1 segmentation_postprocessor_gl.cc:378] NONE activation function chosen on GPU 193cd492c0b2a967a4912923278c1ac367d05d47.js:1086:8
WebGL warning: drawArraysInstanced: Tex image TEXTURE_2D level 0 is incurring lazy initialization.
WebGL warning: readPixels: Format and type RED/FLOAT incompatible with this R32F attachment. This framebuffer requires either RGBA/FLOAT or getParameter(IMPLEMENTATION_COLOR_READ_FORMAT/_TYPE) RGBA/FLOAT. 29
After reporting 32, no further warnings will be reported for this WebGL context.
Source map error: Error: request failed with status 404
Resource URL: https://www.gstatic.com/alkali/608912838d92ccd01c3273ef6b42fff033135226.css
Source Map URL: mediapipestudio-theme.css.map
tyrmullen commented 1 year ago

This is a related issue to https://github.com/googlesamples/mediapipe/issues/159. MPMask elements generated with GPU delegate are not yet supported on all platforms. We're working on patches for these issues now, but in the meantime, try using a CPU delegate-- that will be slower but should hopefully work in the meantime.

annysmolyan commented 1 year ago

For me it doesn't work in Mozilla even with CPU

kuaashish commented 1 year ago

@annysmolyan,

We have observed that image segmentation is functioning as expected in the Android Chrome Browser. Nevertheless, in the Mozilla browser, we cannot guarantee the functionality of the demo, as our testing has primarily focused on the browsers specified in the documentation.

We kindly request you to initiate a new feature request to include support for the Mozilla Browser, highlighting the potential benefits that the community could derive from such an enhancement. Furthermore, we intend to execute the demo on supported browsers and kindly request that you notify us if similar issues are encountered.

Moreover, we would appreciate your confirmation regarding the closure of this issue and the necessary status update, should it no longer be a concern from your end.

opusonline commented 1 year ago

Is there any way to detect GPU support for the current browser to set the correct options for delegate CPU/GPU?

Something like this would be very helpfull:

ImageSegmenter.getSupportedProcessingUnits() // returns a list like ['CPU', 'GPU']

Thanks!

github-actions[bot] commented 1 year ago

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

github-actions[bot] commented 1 year ago

This issue was closed due to lack of activity after being marked stale for past 7 days.

google-ml-butler[bot] commented 1 year ago

Are you satisfied with the resolution of your issue? Yes No