Cannot run `@mediapipe/face_mesh` in web worker

[mattrossman]

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): Yes, but n/a
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Windows 10
• MediaPipe version: n/a, face_mesh version 0.4.1629159166
• Bazel version: n/a
• Solution (e.g. FaceMesh, Pose, Holistic): FaceMesh
• Programming Language and version ( e.g. C++, Python, Java): Javascript

Describe the expected behavior:

I would like to run @mediapipe/face_mesh in a web worker to avoid performance issues with other code on the main thread.

Standalone code you may have used to try to get what you need :

I started writing a minimal demo with the assumption that I could make predictions from an ImageData object, as was the case with the TF.js distribution of face-landmarks-detection: https://glitch.com/~mediapipe-facemesh-worker-issue

However I soon realized that according to the mediapipe solution's type definitions, the only accepted input types are HTML elements:

/**
 * We support several ways to get image inputs.
 */
export type InputImage = HTMLVideoElement | HTMLImageElement | HTMLCanvasElement;

None of these types are available in the worker scope (no DOM access). Additionally, searching for "window" in the JS build reveals a reliance on things like window.location.pathname which is also unavailable to workers.

Other info / Complete Logs :

A related issue was resolved with the Pose solution earlier in the year: #1509

[AlexShafir]

I found a way for web worker:

1. Beatify face_mesh.js
2. Add this code after face_mesh.js line 1632: d.D = d.h.GL.currentContext.GLctx (after d.h.GL.makeContextCurrent(k);)
3. Put your worker file in the same folder with face_mesh.js
4. For every message you post to worker, do not post new messages before you receive result. This is under assumption of vanilla js project.

holistic is fixed in this way too (d.C = d.h.GL.currentContext.GLctx; and line 993)

Working vanilla js example: https://alexshafir.github.io/Sensoria/ Source: https://github.com/AlexShafir/Sensoria You need to rotate camera on the right in (+X & +Y) direction to see face mesh in 3D.

As for your question about input, I post ImageBitmap and it works.

[mattrossman]

@AlexShafir Your demo doesn't work for me, what browser are you using? I am testing in Chrome and Firefox.

Here is the error I see in Chrome:

The other problem is that Firefox doesn't fully support OffscreenCanvas which the image gets painted to before prediction, so the ImageBitmap can't be used there:

Lastly, I am curious how you determined that ImageBitmap works as an input, since it is not listed as an input in the type definitions? And can you explain why you added the d.C = d.h.GL.currentContext.GLctx line?

[AlexShafir]

@mattrossman I am testing on Chrome 94. Just checked rn and it works. Note that on old PC my demo is stuck. I do not consider Firefox, so cannot help there.

How I understood that ImageBitmap works as input - see comment by @tyrmullen. Mediapipe is in alpha -> lots of inconsistencies including type defs.

How I figured why to add said line: that was classic "missing field" bug (I fix such stuff daily, but on Android).

[mattrossman]

I tested in Chrome 94 as well on both MacOS and Window 10, both machines just show an error in the console.

I didn't know mediapipe is alpha, that's good to know. I will avoid using it in my project due to the difficulties I've faced using it versus the TF.js package.

[AlexShafir]

I tested on both Windows (my own machine) and MacOs (3rd party machine), it worked there.

Oh, I remembered error you talked about. When Mediapipe does not detect face, it outputs 0 instead of result and halts - I did not handle this case. It is a known bug #2434.

For my pet project Mediapipe still lacks Iris tracking, so I chose to move with @tensorflow/face-landmarks-detection. It is also alpha stage, but due to Tensorflow automatically selects backend (cpu, WebGl, WebGpu, wasm/simd/threading).

[tyrmullen]

To make tracking easier, let's continue discussion on #2506 and close this issue as a duplicate. Currently running any JS Solution on a worker thread is not fully supported, but that's something we'd like to allow for.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No