Run Holistic in a Web Worker

[AmitMY]

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Chrome
• MediaPipe version: ^0.4.1628005088
• Solution (e.g. FaceMesh, Pose, Holistic): Holistic
• Programming Language and version ( e.g. C++, Python, Java): Javsacript

Describe the expected behavior: In order to run models on the web without the browser hanging (i.e. without clogging the main thread), it is recommended to use a web worker.

Like other image models, where one can communicate with a web worker, I am trying to get that to work with Holistic.

---

Problems:

1. Loading path is not respected

Despite setting up the model this way:

model = new Holistic({locateFile: (file) => `/assets/models/holistic/${file}`});

While the js files are coming from the correct path, the data/wasm files do not:

(if performed on the main thread, it performs as expected)

2. Passing an image as an OffscreenCanvas fails

It is not possible to pass a HTMLVideoElement, or HTMLCanvasElement to a web worker, so instead, at first I take the video frame to a canvas and send its ImageData instead.

Then, in the web worker, I construct an OffscreenCanvas and send that to the model.

The resulting error is as follows:

Referring to v.canvas in the holistic dist code: https://unpkg.com/@mediapipe/holistic@0.4.1628005088/holistic.js

[mhays-google]

Hey Amit, I don't see anything wrong right off the bat. But the initiator for the requests to the data and wasm files is different than the js files. locateFile actually takes two parameters. Can you do me a favor and check to see if it's just giving you weird signatures (weirder than what you've posted already, that is)?

    this.holisticSolution = new holistic.Holistic({
      locateFile: (path, base) => {
        console.log('path', path, 'base', base);
        return `${base}/${path}`;
      }
    });

And good job with the OffscreenCanvas! We should be supporting that properly in the near future, since a lot of us want to use worker threads.

[tyrmullen]

Just to chime in, reading video on the main thread and sending it over to the worker is definitely the right overall approach for now, although I had a few additional pointers:

• Since you're using Chrome, you can get much better performance by avoiding ImageData, as ImageData forces our data to be transferred on the CPU. ImageBitmap should be used instead, so all resources stay on the GPU.

• You shouldn't need to pass the image as an OffscreenCanvas-- ImageBitmap should suffice, since it's just being used to create a WebGLTexture anyways-- if you want to use ImageData instead, you can just cast it to another type, like HTMLVideoElement (although you might need to give it 'videoWidth' and 'videoHeight' properties, set equal to 'width' and 'height', respectively).

• However, you're right that OffscreenCanvas is what's missing here-- the JS Solutions framework is currently hard-coded to create an HTMLCanvasElement for internal use, so a patch in MediaPipe code would be necessary to avoid the "canvas undefined" error you're seeing.

[AmitMY]

@mhays-google Here is the console output for path and base:

@tyrmullen Thanks, I am now doing const bitmap = await createImageBitmap(video); and passing that to the worker. I could either draw it on an OffscreenCanvas or just pass it directly to Holistic, however, both are not currently supported because, like you said, it creates a canvas which is not available in the worker context.

[tvanier]

@AmitMY for the loading path, maybe try the URL constructor with origin as second argument?

model = new Holistic({
  locateFile: (file) =>
    new URL(`/assets/models/holistic/${file}`, globalThis.location.origin).toString()
});

[tvanier]

and +1 for a lower-level API (ImageBitmap) with support for web workers (I use selfie-segmentation) where should I upvote? ;-)

[AlexShafir]

I found a way for web worker:

1. Beatify holistic.js
2. Add this code after holistic.js line 993: d.C = d.h.GL.currentContext.GLctx (after d.h.GL.makeContextCurrent(k);)
3. Put your worker file in the same folder with holistic.js
4. For every message you post to worker, do not post new messages before you receive result.

This is under assumption of vanilla js project.

face_mesh is fixed in this way too (d.D = d.h.GL.currentContext.GLctx; and line 1632)

Working vanilla js example: https://alexshafir.github.io/Sensoria/ Source: https://github.com/AlexShafir/Sensoria You need to rotate camera on the right in (+X & +Y) direction to see face mesh in 3D.

[AmitMY]

Thanks for the workaround, @AlexShafir However, @sgowroji, I still argue for a native solution in the package, and support for OffscreenCanvas or better yet, an ImageBitmap

[AlexShafir]

@AmitMY In my demo I pass ImageBitmap directly to holistic.js index.js:

createImageBitmap(videoElement).then((bitmap) => {
        holisticWorker.postMessage(bitmap, [bitmap]) // transferable
    })

holistic_worker.js:

onmessage = (event) => {
    holistic.send({image: event.data})
}

[AlexShafir]

For me bummer is that even after workaround holistic.js is ultra slow on high-end laptop, so I switched to face_mesh.js which shows a lot better performance.

[AmitMY]

new URL(`/assets/models/holistic/${file}`, globalThis.location.origin).toString()

Thanks, @tvanier , this does not change the result, unfortunately. The path is still not correctly resolved

[mhays-google]

Workers are on our roadmap, but I unfortunately don't have a window to provide this. ImageBitmap is already an output format where supported (i.e., Chrome), but if I hear you right, you would like an input format of ImageBitmap?

On Mon, Sep 27, 2021 at 1:33 AM Amit Moryossef @.***> wrote:

new URL(/assets/models/holistic/${file}, globalThis.location.origin).toString()

Thanks, @tvanier https://github.com/tvanier , this does not change the result, unfortunately. The path is still not correctly resolved

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/2506#issuecomment-927650052, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFJ4TTI4U3SKZ6LHC23UEAT4NANCNFSM5DQFYGDQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[tyrmullen]

ImageBitmap is already an input format as well, as demonstrated by @AlexShafir .

[AmitMY]

@AlexShafir Your changes seem to be minimal. If it's not too much trouble, could you please contribute a fix to this repository?

[AlexShafir]

@AmitMY Source code for Js Solutions is not released, see #1408 (basically wasm does not fit nicely into Bazel). So one cannot make PR.

My solution is just a workaround. To make it complete, one needs to add proper path handling - at least.

[AmitMY]

(Can confirm, 0.4.1633559476 did not fix problems 1 and 2)

[tyrmullen]

As mhays-google@ mentioned, this is not an issue we have an ETA for at this time.

As an aside-- if the purpose of switching to a worker thread is to improve performance, that is probably not going to happen here. Most of the processing taking place is for the ML inference, and the majority of the remainder is for rendering. Both of these are going to be occurring GPU-side, and (at least on Chrome) there's a single shared GPU process for all tabs and threads, so you're unlikely to get any parallelization benefits off the bat. There are other reasons why workers can be useful, of course, but for pure performance, they're unlikely to help here.

Note that this also will make it tricky to use workers to unblock the main thread currently, since any CPU/GPU sync (any time we "wait for GPU operations to finish", like a gl.readPixels call for example) will result in us waiting for the worker thread's GPU work to finish as well.

TL;DR: For pure speed GPU > CPU, but for parallelization CPU > GPU. Workers can occasionally help, but won't be easy/straightforward given how GPU-reliant we are by default.

[AmitMY]

Thanks @tyrmullen I'll just note that the largest, and most apparent, and annoying main thread block is loading the holistic model, not inference, in my case. When it is loading, nothing else in my app works, and that sucks because it happens at a time when people do want to interact with the app. That's my main use case here, and secondary is inference block.

[tyrmullen]

No problem! And yes, that makes sense, and workers could probably help for that-- I'd be curious to see just how helpful they can be on this front (they should allow us to make those "wait for GPU" ops be non-blocking, but would still impact the GPU queue).

One thing to note is that loading the model data is likely taking very little time itself (or it'd show as CPU processing time), but rather the initialization of the ML inference engine is quite expensive. I suspect the concentration of loading time you're seeing will be spread between two spikes: (a) one occurring during the MediaPipe graph initialization and (b) one occurring during the first frame processed (the first 'send' call), probably right afterwards. That's because shader program instantiation is likely the main cost here, and WebGL lazily enforces shader initialization, so while a lot of work is done up front, the process doesn't complete until the first frame is rendered. If your performance profiling matches this pattern, then this is almost certainly the culprit.

[AmitMY]

Can confirm this is still an issue in version 0.5.1635989137

[fgsreally]

My personal feeling is, pose Initialize () does not complete the real initialization. One trick is to send a non empty video element in advance before playing the video. However, a moment of Caton cannot be avoided in any way (including worker).

I hope there is a way to reduce the impact of real initialization on the window. I mean the blocking of interface rendering rather than time-consuming. Even if it takes more time, it is worth it (because there are ways to cover up the disadvantage of time-consuming, but there is no way to solve the blocking of rendering

Web worker is not the key

[AmitMY]

Any update on this from the maintainers? If the mediapipe JS code will be open source, I'm sure someone (perhaps even I) could solve it.

[tvanier]

I second @AmitMY , it would be great if the mediapipe JS could run in a worker, and send resolving with the actual results. Would fit nicely with the insertable streams API!

// worker.js
const holistic = new Holistic(/* ... */)
const canvas = new OffscreenCanvas(/* ... */)

const transformer = new TransformStream({
    async transform(videoFrame, controller) {
        const results = await holistic.send({ image: videoFrame })
        // draw results onto canvas and
        const newFrame = new VideoFrame(canvas)
        controller.enqueue(newFrame)
    }
})

self.onmessage = (message) => {
    // main thread transfers a MediaStreamTrackProcessor.readable and MediaStreamTrackGenerator.writable
    const { readable, writable } = message.data
    readable.pipeThrough(transformer).pipeTo(writable)
}

[w-okada]

I created the webworker demo using the mediapipe models (face, hand, pose). https://d3iwgbxa9wipu8.cloudfront.net/P01_wokers/t18_mediapipe-mix/index.html

(Note: some part of process is guess-work.) The repo is linked at the upper right icon.

[AmitMY]

@w-okada Seems like you are using @tensorflow-models and not mediapipe directly. Please correct me if I am mistaken. While that can work, this issue is specifically about running mediapipe models.

[w-okada]

@AmitMY I use the model listed at the page below. https://google.github.io/mediapipe/solutions/models.html

[KTRosenberg]

Has anyone found a workaround for the path-resolution on web workers?

[ButzYung]

@tyrmullen Running mediapipe on web worker may not do much on high-end devices, but it does make a difference on lower-end devices. Say if your device can only run mediapipe at below 20fps, if you run everything on the main thread, your graphics rendering will also be stuck at the same frame rate (the cause can be the CPU overhead used by the mediapipe models, or it's the way the GPU resources are shared in the main thread). However, if you put mediapipe on a web worker, you can keep your main thread rendering at 30fps or higher with very little impact on the inference speed of the mediapipe models.

I have a web app that loads mediapipe on web workers. I put a stress test on a medicore smart phone, and although the inference speed is just around 5fps, the 3D graphics on the main thread is still at 30fps.

https://twitter.com/butz_yung/status/1519365509128417280/

[AmitMY]

@ButzYung how did you manage to run it in a worker? Any trick we could all use?

[ButzYung]

@AmitMY I took a similar approach as @AlexShafir did before by editing the beautified version of holistic.js (BTW I tried his method on 0.5.1635989137 some hours ago but it didn't seem to work anymore). Our edits are basically on the same location.

            case 6:
                e.h = p.h;
// hack for worker
/*
                a.l = new OffscreenCanvas(1,1);
                a.h.canvas = a.l;
                g = a.h.GL.createContext(a.l, {
                    antialias: !1,
                    alpha: !1,
                    aa: "undefined" !== typeof WebGL2RenderingContext ? 2 : 1
                });
                a.h.GL.makeContextCurrent(g);
                p.g = 4;
                break;
*/
            case 7:
// hack for worker
                a.l = ((typeof Document !== 'undefined') && document.createElement("canvas")) || new OffscreenCanvas(1,1);

My approach is to basically comment out the whole case 6, and do everything worker and non-worker on case 7. And on the first line, I edit it to use OffscreenCanvas whenever web worker is used.

As for the path issue, put the worker file in the same folder with holistic.js, just like AlexShafir's version did.

[tyrmullen]

@ButzYung That's what I meant by saying workers wouldn't help pure performance of the ML inference-- that the 5fps framerate would not increase (and might potentially decrease, depending). Workers can definitely help unblock the main thread, for fast and slow devices alike, but when it's GPU it's usually less about overhead and more about synchronization and waiting; Holistic has to wait on CPU for GPU to finish for every frame, while most pure rendering doesn't (and shouldn't). So that's why I said running on workers could definitely help overall app behavior, but not the inference framerate (or initialization/load time) [all this is assuming inference is running on GPU-- on CPU you get actual gains, often 2-3x factor, since the threads are relatively independent].

However, all that being said, I'm a bit surprised/impressed that you see so little impact to your main rendering thread given that I'd imagine the GPU should be quite fully utilized by the ML inference (if that part can only run at 5fps-- which is also quite low for GPU ML, even on moderate devices); that makes me suspect the inference is running on CPU and not GPU. Were you testing on an iPhone by any chance? (we force CPU inference there). I guess one way to check too would be to look at the Chrome Developer console performance traces and see if the worker looks like it's using a lot of WebGL calls at the end of the call stacks?

[KTRosenberg]

@AmitMY I took a similar approach as @AlexShafir did before by editing the beautified version of holistic.js (BTW I tried his method on 0.5.1635989137 some hours ago but it didn't seem to work anymore). Our edits are basically on the same location.

            case 6:
                e.h = p.h;
// hack for worker
/*
                a.l = new OffscreenCanvas(1,1);
                a.h.canvas = a.l;
                g = a.h.GL.createContext(a.l, {
                    antialias: !1,
                    alpha: !1,
                    aa: "undefined" !== typeof WebGL2RenderingContext ? 2 : 1
                });
                a.h.GL.makeContextCurrent(g);
                p.g = 4;
                break;
*/
            case 7:
// hack for worker
                a.l = ((typeof Document !== 'undefined') && document.createElement("canvas")) || new OffscreenCanvas(1,1);

My approach is to basically comment out the whole case 6, and do everything worker and non-worker on case 7. And on the first line, I edit it to use OffscreenCanvas whenever web worker is used.

As for the path issue, put the worker file in the same folder with holistic.js, just like AlexShafir's version did.

So you need the offscreen canvas? It would be nice to have a complete runnable example.

[ButzYung]

@tyrmullen I tested it on an android phone with Snapdragon 845. I suppose it is running on GPU (I doubt it will be anything more than 1fps if it runs on CPU), but unfortunately I can't check the web console with my current setup. In fact, I tested it with non-worker case as well, and the inference speed is almost the same (5-7fps) but the 3D rendering speed drops significantly to around the same fps as the inference.

As to why 3D rendering speed can be maintained when ML model is running on web worker, we make the wrong assumption that ML inference can fully utilize the GPU, but in fact it cannot. I tested my app on a medicore PC (GTX1650) with the inference speed being around 25fps average, but the ML inference and 3D rendering combined is using less than 1/3 of the the total GPU speed. There should be enough GPU resource to improve the inference speed, but for some reasons the browser just leaves it idle. That's why the 3D rendering speed in the main thread can be maintained and they won't affect each other (as long as neither side is using too much GPU resource).

PS: To anyone curious, you can test my web app here.

worker enabled: https://sao.animetheme.com/XR_Animator.html

worker disabled: https://sao.animetheme.com/XR_Animator.html?ML_worker_disabled=1

[tyrmullen]

It's been a while since I've tried anything on Android Chrome (I'm assuming this is Chrome), so that's very good to know! I wonder if that's because the repeated CPU/GPU sync-ing gives it some breathing room, or if the browser is playing a more active role in the throttling. Perhaps one way to test the first would be to try with Segmentation instead of Holistic and see if that changes things, since Segmentation doesn't have any CPU/GPU syncs (Holistic has one in order to bring back the GPU ML to CPU so the pose landmarks can be handled CPU-side)?

With the PC experiments you'll have to be a little bit more careful, since if you're using webcam, the browser definitely caps the framerate to 25-30fps, so it sounds like it could possibly be that you're just hitting that framerate limit, and won't be able to really test GPU utilization fully. Also, your PC's overall GPU situation should definitely always be underutilized, since there should be a single shared-across-all-tabs Chrome GPU process, so if you look at more fine-grained device metrics, I'd expect it to look like one GPU thread is being utilized a ton (for all the 3D rendering and ML inference), while the other threads may likely be sitting idle or not doing much. But I haven't checked out how utilized that one thread is (and haven't really been running on Android lately), and at least on Android with Snapdragon 845, it sounds like perhaps surprisingly the answer is "not fully utilized at all", which runs counter to a lot of our earlier PC experiments.

In any case, definitely something interesting to look into and keep in mind.

[Justinyu1618]

@ButzYung Did you have to edit any other part of the code to get this to work? When I try to initialize the model in a web worker, it tries to run importScripts on the tflite file, which errors out, in addition to a bunch of other issues because there's no document or window.

[ButzYung]

@Justinyu1618 Yeah you have to edit holistic.js, as shown in my previous comment. Errors should be gone after that.

[Wei-1]

@ButzYung 's editing actually work, here is a simpler working example: https://github.com/Wei-1/holistic-website-test

[AmitMY]

@mhays-google as you may see there is a solution - could you please implement this in the public npm release instead of people needing to edit themselves?

having it running in a worker thread is essential for performant and fluent user experience

[davideloba]

thanks to @Wei-1 and @ButzYung this is my test with patched facemesh

[dev-logarithm]

any solution for this issue ?

[x6ud]

I found that pose's web worker support implement is incomplete. Here is my solution without modifying the origin file:

1. Copy /node_modules/@mediapipe/pose to a static path, in my case is /assets

2. Create a classic web worker (rather than module type, otherwise you can't use importScripts) const worker = new Worker('/assets/detect-pose.worker.js', {type: 'classic'});

3. In detect-pose.worker.js

const POSE_CONFIG = {
    locateFile(path, prefix) {
        return '/assets/@mediapipe/pose/' + path;
    }
};
self.createMediapipeSolutionsWasm = POSE_CONFIG;
self.createMediapipeSolutionsPackedAssets = POSE_CONFIG;
importScripts(
    '/assets/@mediapipe/pose/pose.js',
    '/assets/@mediapipe/pose/pose_solution_packed_assets_loader.js',
    '/assets/@mediapipe/pose/pose_solution_simd_wasm_bin.js',
);

(function () {
    let pose;
    let detectPoseResults;
    self.onmessage = async function (e) {
        if (!pose) {
            pose = new Pose(POSE_CONFIG);
            pose.setOptions({
                selfieMode: false,
                modelComplexity: 2,
                smoothLandmarks: false
            });

            const solution = pose.g;
            const solutionConfig = solution.g;
            solutionConfig.files = () => []; // disable default import files behavior
            await pose.initialize();
            solution.D = solution.h.GL.currentContext.GLctx; // set gl ctx

            // load data files
            const files = solution.F;
            files['pose_landmark_heavy.tflite'] = (await fetch('/assets/@mediapipe/pose/pose_landmark_heavy.tflite')).arrayBuffer();
            files['pose_web.binarypb'] = (await fetch('/assets/@mediapipe/pose/pose_web.binarypb')).arrayBuffer();

            // set callback
            pose.onResults(function onResults(results) {
                detectPoseResults = results;
            });
        }
        pose.reset();
        const bitmap = e.data;
        await pose.send({image: bitmap});

        postMessage(detectPoseResults);
    };
})();

4. To use

   function detectPose(image: HTMLImageElement) {
   return new Promise<PoseResults>(async function (resolve) {
       worker.onmessage = function (e) {
           resolve(e.data);
       };
       worker.postMessage(await createImageBitmap(image));
   });
   }

The problem is that initialize() is trying to load the data files with importScripts in web worker scope, and forgot to set gl to it's context. So just disable the original file importing and load the required files yourself.

[ayoub-root]

thanks to @x6ud

the code is not working in react , i made some changes

• create the worker file in assets folder

/* eslint-disable no-undef */

/* eslint-disable no-restricted-globals */
const POSE_CONFIG = {
  locateFile(path, prefix) {
    return "/pose/" + path;
  },
};
self.createMediapipeSolutionsWasm = POSE_CONFIG;
self.createMediapipeSolutionsPackedAssets = POSE_CONFIG;
importScripts(
  "/pose/pose.js",
  "/pose/pose_solution_packed_assets_loader.js",
  "/pose/pose_solution_simd_wasm_bin.js"
);
let pose;

(async function () {
  if (!pose) {
    console.log("start pose");
    pose = new Pose(POSE_CONFIG);
    pose.setOptions({
      selfieMode: false,
      modelComplexity: 0,
      smoothLandmarks: false,
    });
    const solution = pose.g;
    const solutionConfig = solution.g;
    solutionConfig.files = () => []; // disable default import files behavior
    await pose.initialize();
    solution.D = solution.h.GL.currentContext.GLctx; // set gl ctx

    // load data files
    const files = solution.F;
    files["pose_landmark_lite.tflite"] = (
      await fetch(" /pose/pose_landmark_lite.tflite")
    ).arrayBuffer();
    files["pose_web.binarypb"] = (
      await fetch(" /pose/pose_web.binarypb")
    ).arrayBuffer();

    // set callback
    pose.onResults(function onResults(results) {
      postMessage(results);
    });
  }
  self.onmessage = async function (e) {
    // const bitmap = e.data;
    await pose.send({ image: e.data });
  };
  pose.reset();
})();

• my pose folder was in /public (i'm working with react)
• i updated this instruction in pose.js (worker doesn't support "document")

(navigator.userAgent.includes("Mac") && "ontouchend" in document),

=>

(navigator.userAgent.includes("Mac")

• i noticed that worker is very slow , (5s and more of latency between the original video"camera' and the generated one )

[AmitMY]

I have tried @x6ud's solution with @mediapipe/holistic

It is very promising, but not yet fully working. It seems to be running the pose estimation, but only returns {image: ImageBitmap, multiFaceGeometry: Array(0)} instead of the full object with all the poses.

Code: https://github.com/sign/translate/pull/28 Demo: https://translate-sign-mt--pr28-holistic-worker-nzqu5dt3.web.app/playground

Open console: it will print the results for every frame.

[x6ud]

For more details, the pose.js without minification should be like:

async function fetchFile(solution, url) {
    const files = solution.F;
    if (url in files) {
        return files[url];
    }
    url = solution.locateFile(url, '');
    const data = (await fetch(url)).arrayBuffer();
    return files[url] = data;
}

class Pose {
    async initialize() {
        const solution = this.g;
        const config = solution.g;
        let files;
        if (config.files) {
            if (typeof config.files === 'function') {
                let options = solution.j;
                files = config.files(options);
            } else {
                files = config.files;
            }
        } else {
            files = [];
        }
        if (typeof window === 'object') {
            window.createMediapipeSolutionsWasm = {locateFile: solution.locateFile};
            window.createMediapipeSolutionsPackedAssets = {locateFile: solution.locateFile};
            const dataFiles = files.filter(file => file.data);
            const scriptFiles = files.filter(file => !file.data);
            const fetchDataFilesPromise = Promise.all(dataFiles.map(async file => {
                const dataPromise = await fetchFile(solution, file.url);
                if (file.path) {
                    solution.overrideFile(file.path, await dataPromise);
                }
                return dataPromise;
            }));
            const importScriptsPromise = Promise.all(
                scriptFiles.map(async file => {
                    if (file.simd === useSimd) {
                        const basePath = solution.S;
                        await insertScriptTagToDocument(solution.locateFile(file.url, basePath));
                    }
                }))
                .then(async () => {
                    // pose_solution_simd_wasm_bin.js and pose_solution_packed_assets_loader.js will set those funtions to window
                    const wasmCtx = await window.createMediapipeSolutionsWasm(window.createMediapipeSolutionsPackedAssets());
                    solution.h = wasmCtx;
                });
            const fetchGraphPromise = (async () => {
                if (config.graph?.url) {
                    await fetchFile(solution, config.graph.url);
                }
            })();
            await Promise.all([importScriptsPromise, fetchDataFilesPromise, fetchGraphPromise]);
            const canvas = solution.l = document.createElement('canvas');
            let gl = canvas.getContext('webgl2', {});
            if (!gl) {
                gl = canvas.getContext('webgl', {});
            }
            if (!gl) {
                alert('error');
                return;
            }
            solution.D = gl;
            const wasmCtx = solution.h;
            wasmCtx.canvas = canvas;
            wasmCtx.createContext(canvas, true, true, {});
        } else {
            // run in web worker
            if (typeof importScripts !== 'function') {
                throw Error('...');
            }
            const fileUrls = files.filter(file => file.simd === useSimd).map(file => {
                const basePath = solution.S;
                return solution.locateFile(file.url, basePath);
            });
            // this will cause an error when importing .tflite file
            importScripts.apply(null, fileUrls);
            await createMediapipeSolutionsWasm(Module);
            const canvas = solution.l = new OffscreenCanvas(1, 1);
            const wasmCtx = solution.h;
            wasmCtx.canvas = canvas;
            const gl = wasmCtx.createContext(canvas, {antialias: false, alpha: false});
            wasmCtx.GL.makeContextCurrent(gl);
        }
        // ...
    }

    async send(inputs, at) {
        const solution = this.g;
        if (!solution.inputs) {
            return;
        }
        const timestamp = 1e3 * (at == null ? performance.now() : at);
        await this.initialize();
        const packetDataList = new PacketDataList();
        for (let key of Object.keys(inputs)) {
            const input = inputs[key];
            const info = solution.inputs[key];
            switch (info.type) {
                case 'video': {
                    const cachedTexture = this.m;
                    let texture = cachedTexture[info.stream];
                    if (!texture) {
                        const wasmCtx = solution.h;
                        const gl = solution.D;
                        texture = cachedTexture[info.stream] = new Texture(wasmCtx, gl);
                        const glTexture = texture.l;
                        if (glTexture === 0) {
                            texture.l = wasmCtx.createTexture();
                        }
                    }
                    let width, height;
                    if (typeof HTMLVideoElement !== 'undefined' && input instanceof HTMLVideoElement) {
                        width = input.videoWidth;
                        height = input.videoHeight;
                    } else if (typeof HTMLImageElement !== 'undefined' && input instanceof HTMLImageElement) {
                        width = input.naturalWidth;
                        height = input.naturalHeight;
                    } else {
                        // you might want to modify this for different transferable objects
                        width = input.width;
                        height = input.height;
                    }
                    const tex = {glName: texture.l, width, height};
                    const gl = texture.g;
                    gl.canvas.width = tex.width;
                    gl.canvas.height = tex.height;
                    gl.activeTexture(gl.TEXTURE0);
                    const wasmCtx = texture.h;
                    wasmCtx.bindTexture2d(texture.l);
                    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, input);
                    gl.bindTexture2d(0);
                }
                break;
                case 'detections': {
                    // ...
                }
                break;
            }
            // ...
        }
        // ...
    }
}

Other solutions may be different.

In addition, you may need to use other transferable object to avoid unnecessary data type conversion

[prestonbourne]

@tyrmullen Running mediapipe on web worker may not do much on high-end devices, but it does make a difference on lower-end devices. Say if your device can only run mediapipe at below 20fps, if you run everything on the main thread, your graphics rendering will also be stuck at the same frame rate (the cause can be the CPU overhead used by the mediapipe models, or it's the way the GPU resources are shared in the main thread). However, if you put mediapipe on a web worker, you can keep your main thread rendering at 30fps or higher with very little impact on the inference speed of the mediapipe models.

I have a web app that loads mediapipe on web workers. I put a stress test on a medicore smart phone, and although the inference speed is just around 5fps, the 3D graphics on the main thread is still at 30fps.

https://twitter.com/butz_yung/status/1519365509128417280/

@ButzYung Are you able to produce an example snippet to demo this? I'm currently working with the hand landmarking model and trying to use a worker to run the detection in the background

const worker = new Worker('worker', {type: 'module'});

class Foo{
 constructor(){/* .... */}
 ...

  updateLandmarks(){

    const nowInMs = Date.now();
    worker.postMessage(this.$videoEl)

    window.requestAnimationFrame(this.updateLandmarks.bind(this));

  }
}

then in my worker.js file (i'm aware that I am not sending my results back)

import handLandmarker from "./landmarker"

self.onmessage = (e) => {

 handLandmarker.detectForVideo(e.data.$videoEl, nowInMs)

 self.postMessage(results)
}

I encounter the following error:

vision_bundle.js:1 Uncaught TypeError: Failed to execute 'importScripts' on 'WorkerGlobalScope': Module scripts don't support importScripts().
    at o (vision_bundle.js:1:466026)
    at Array.map (<anonymous>)
    at r (vision_bundle.js:1:466152)
    at createTaskRunner (vision_bundle.js:1:467669)
    at TaskRunner.createInstance (vision_bundle.js:1:467909)
    at VisionTaskRunner.createVisionInstance (vision_bundle.js:1:472322)
    at b.createFromOptions (vision_bundle.js:1:553534)
    at index.js?t=1683573253005:8:45

Perhaps it may pertain to how I'm loading and exporting the model:

import {HandLandmarker, FilesetResolver} from "@mediapipe/tasks-vision"
import model from './hand_landmarker.task'

const vision = await FilesetResolver.forVisionTasks(
 // path/to/wasm/root
 "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

const handLandmarker = await HandLandmarker.createFromOptions(
   vision,
   {
     baseOptions: {
       modelAssetPath: model
     },
     numHands: 1,
     runningMode: "VIDEO"
   });

export default handLandmarker;

[ButzYung]

@Saintpreston I haven't tried the new tasks version of the pose landmarker since it is still WIP, but judging from the error message, a wild guess is that you need to load the scripts in the non-module mode of web worker I suppose?

[Wei-1]

Just btw, I am only able to get the trick that @ButzYung shared work up to version: 0.5.1635989137 For version: 0.5.1675471629 that was updated 3 months ago, I am not able to make it run in the worker. If @Saintpreston you are trying now, maybe use the version 0.5.1635989137, which should be easier to start.

[ButzYung]

Just btw, I am only able to get the trick that @ButzYung shared work up to version: 0.5.1635989137 For version: 0.5.1675471629 that was updated 3 months ago, I am not able to make it run in the worker. If @Saintpreston you are trying now, maybe use the version 0.5.1635989137, which should be easier to start.

0.5.1675471629 works for me on worker with the same trick. The issue is that the output result of world_landmarks has changed from .ea to .za

[kuaashish]

@AmitMY,

MP Tasks for Web support Web Workers. However, We are working towards publish the Holistic solutions Asap from our end And let you know once becomes available.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.