Run Holistic in a Web Worker

[AmitMY]

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Chrome
• MediaPipe version: ^0.4.1628005088
• Solution (e.g. FaceMesh, Pose, Holistic): Holistic
• Programming Language and version ( e.g. C++, Python, Java): Javsacript

Describe the expected behavior: In order to run models on the web without the browser hanging (i.e. without clogging the main thread), it is recommended to use a web worker.

Like other image models, where one can communicate with a web worker, I am trying to get that to work with Holistic.

---

Problems:

1. Loading path is not respected

Despite setting up the model this way:

model = new Holistic({locateFile: (file) => `/assets/models/holistic/${file}`});

While the js files are coming from the correct path, the data/wasm files do not:

(if performed on the main thread, it performs as expected)

2. Passing an image as an OffscreenCanvas fails

It is not possible to pass a HTMLVideoElement, or HTMLCanvasElement to a web worker, so instead, at first I take the video frame to a canvas and send its ImageData instead.

Then, in the web worker, I construct an OffscreenCanvas and send that to the model.

The resulting error is as follows:

Referring to v.canvas in the holistic dist code: https://unpkg.com/@mediapipe/holistic@0.4.1628005088/holistic.js

[rummik]

@kuaashish Has this been fixed? It looks like github-actions just mistakenly closed it and it wasn't reopened afterwards, which might make it a bit difficult to be notified when the issue is actually resolved 😅

[Wei-1]

This is no resolved. But if the GitHub action will close any issue after 7 days without action, it will be closed after 7 days if we reopen it anyway?

[AmitMY]

Release v0.10.5 claims:

Don't access "document" in WebWorker

but @mediapipe/tasks-vision still doesn't support holistic, and holistic package was not updated

[lc-spxl]

Holistic is not part of tasks-vision yet. Any plan for release ? Thank you

[kyasuda2003]

MP Tasks for Web support Web Workers. However, We are working towards publish the Holistic solutions Asap from our end And let you know once becomes available.

Maybe I am bit outdated..

The usage from your latest doc

const vision = await FilesetResolver.forVisionTasks(
  // path/to/wasm/root
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
objectDetector = await ObjectDetector.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath: `https://storage.googleapis.com/mediapipe-tasks/object_detector/efficientdet_lite0_uint8.tflite`
  },
  scoreThreshold: 0.5,
  runningMode: runningMode
});

if as you suggested the latest MP can be run in a worker, then I assume it can only be run with {type: `module`}

but this signature also suggest importScripts must be in place for loading the wasm which is ambiguous to yours. https://github.com/google/mediapipe/blob/master/mediapipe/web/graph_runner/graph_runner.ts#L53

I overloaded the signature to run that in a worker but the performance is less than expected. Is there something you can advise otherwise? kuaashish

[tyrmullen]

@kyasuda2003 The model you're using there has a _uint8 suffix, so it's quantized. This will help it run more efficiently on CPU, but means that it cannot run on GPU, so in isolation it is expected to run much slower than if you were able to use delegate: GPU. I'm not sure why the function importScripts is declared there (perhaps just to get rid of some TS compiler warning/error depending on settings?), but in the call where it's actually used (which looks like it's now in run_script_helper.ts.template, it is supposed to be used only where it actually exists (i.e. from a worker context), so new MP Solutions in general are expected to run from a worker (on non-iOS platforms) as well as on the main/UI thread (on all supported platforms).

I can also confirm that Holistic is not yet an offering for the new MP Solutions, but cannot comment as to release plans for it.

[epleaner]

Bumping this. Has anyone successfully run Holistic in a Web Worker? It's unclear from the earlier comments.

[matiaslopezd]

@epleaner yes with the newest version of Mediapipe, but be aware of the memory. I had some cases of out of the memory in different browser versions. But I think could be my code not cleaning correctly the Offscreencanvas.

Also, check the module bundler you will use considering not all handle workers correctly. Or just importScripts from a CDN.

Edit: the tflite, js and wasm latest versions are not compatible with previous versions

[matiaslopezd]

@kyasuda2003 The model you're using there has a _uint8 suffix, so it's quantized. This will help it run more efficiently on CPU, but means that it cannot run on GPU, so in isolation it is expected to run much slower than if you were able to use delegate: GPU. I'm not sure why the function importScripts is declared there (perhaps just to get rid of some TS compiler warning/error depending on settings?), but in the call where it's actually used (which looks like it's now in run_script_helper.ts.template, it is supposed to be used only where it actually exists (i.e. from a worker context), so new MP Solutions in general are expected to run from a worker (on non-iOS platforms) as well as on the main/UI thread (on all supported platforms).

I can also confirm that Holistic is not yet an offering for the new MP Solutions, but cannot comment as to release plans for it.

I had some cases of out of memory delegating to the GPU compared with the CPU. Maybe but not sure if it is something about cleaning my OffScreenCanvas or not. However, this should be another issue.

[epleaner]

@epleaner yes with the newest version of Mediapipe, but be aware of the memory. I had some cases of out of the memory in different browser versions. But I think could be my code not cleaning correctly the Offscreencanvas.

Also, check the module bundler you will use considering not all handle workers correctly. Or just importScripts from a CDN.

Edit: the tflite, js and wasm latest versions are not compatible with previous versions

Do you have any code snippets or a repository that has the newest version working in a Web Worker?

[Wei-1]

@epleaner yes with the newest version of Mediapipe, but be aware of the memory. I had some cases of out of the memory in different browser versions. But I think could be my code not cleaning correctly the Offscreencanvas. Also, check the module bundler you will use considering not all handle workers correctly. Or just importScripts from a CDN. Edit: the tflite, js and wasm latest versions are not compatible with previous versions

Do you have any code snippets or a repository that has the newest version working in a Web Worker?

Maybe you can try this? https://github.com/Wei-1/holistic-website-test

[epleaner]

Thanks, that does work locally. I'm trying to port this over to NextJS, and oddly I'm getting a whole bunch of window/document is not defined errors using the same holistic.js / Web Worker file in this repository.

[kuaashish]

Hi @AmitMY,

We have upgraded our MediaPipe Legacy Solutions to new MediaPipe solutions, which now include the new Holistic Task API and supports the Web Worker.

Could you please try using the new HolisticLandmarker task APIs, and let us know if the problem persists?

Thank you!!

[AmitMY]

Thanks @kuaashish The guide says "Coming soon" https://developers.google.com/mediapipe/solutions/vision/holistic_landmarker

There's also no information in https://www.npmjs.com/package/@mediapipe/tasks-vision

[kuaashish]

Hi @AmitMY,

We expect to release samples soon, starting with Android, followed by others. Documentation will be available afterwards. Though we can not give an exact date, it is in progress and will be out soon.

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[AmitMY]

(commenting to un-stale the issue, waiting for the solution)