Webworker

[patrick-nurt]

Hi there! Great work on this plugin! Has anybody managed to run this in a webworker?

[thexiroy]

I haven't tried it but it should work. I am assuming you want to use a webworker so that the ui doesn't get blocked because of loading the models or during face detection/recognition? Do keep in mind that you won't be able to access dom elements from a webworker.

[patrick-nurt]

Wanted to use a tensor as input in the worker, to avoid DOM elements, would just transfer the image data to the worker and then transfer the results back m.

[akofman]

I did it and it works, but at the moment tfjs is not compatible with offscreen canvas (https://github.com/tensorflow/tfjs/issues/102) so you don't have access to the GPU from a webworker and the result is more than slow actually ... If your purpose is to get better performances I advice you to have a try with the mtcnn model which is really faster, see the 0.9.0 version of this project.

[Floby]

@akofman Out of curiosity, how did you make it work ?

[shershen08]

I had same experience with tracking.js library. I managed to to put its part to web worker https://github.com/eduardolundgren/tracking.js/issues/99 and completely get rid of the UI lags but code was just for POC

[ScottDellinger]

I did it and it works, but at the moment tfjs is not compatible with offscreen canvas (tensorflow/tfjs#102) so you don't have access to the GPU from a webworker and the result is more than slow actually ... If your purpose is to get better performances I advice you to have a try with the mtcnn model which is really faster, see the 0.9.0 version of this project.

@akofman I also would be interested in how you got it working in a worker, if you're willing to share!

[jeffreytgilbert]

Bumping this for relevance because i'm now working on doing the same thing and with offscreen canvas. My project is here if anyone wants to check out why. I'm rendering a threejs scene and the face detection allows me to use face detection to control the perspective of the 3d scene. Each time the face detection runs at 100ms, the scene janks out briefly because the 3d part runs in about 1ms cpu time over 16ms budget for RAF calls, but the face detection part goes 40ms-100ms so you lose 3 frames at a time, making it look like the rendering is broken. Now I have to find a way to get it under budget for the reasons @thexiroy mentioned

[justadudewhohacks]

It's probably better to ask for help at tfjs regarding how to get this running in a webworker

[jeffreytgilbert]

For those interested, there is a pull request open here which was a bit dated, but I've been working on. The OffscreenCanvas support doesn't appear that involved, but there doesn't appear to be any special consideration for web workers or transferable objects, and those may take longer to integrate in. I did see there is a branch for web workers open, but I haven't been through it.

https://github.com/tensorflow/tfjs-core/pull/1221

[jeffreytgilbert]

@justadudewhohacks tfjs updates is a non-starter. Typescript wont have support for OffscreenCanvas until 3.5.0 and that's not officially released. Even if they do release it, it's currently buggy and TensorFlow wont build against 3.5.0 without ts-ignore hacks. Even if you do the ts-ignore hacks, the resulting build of tensorflow running in face-api.js barf due to those ignored incompatibilities. Models don't work. flattened maps don't work. Whole thing barfs.

So, hacking created and updated tickets for those findings. The tfjs-core update thread above was updated. I also created a typescript issue that can be tracked here: https://github.com/Microsoft/TypeScript/issues/30998

It looks like things on both those projects move pretty quickly, so hopefully this wont fall to the bottom of the thousands of filed issues on the pile and actually get some updates. For now, I will be attempting to create a fake interface in the worker thread which proxies back to the main js with updates and commands. The approach is similar to one that @mizchi took here: https://github.com/tensorflow/tfjs/issues/102#issuecomment-462167706

The difference between @mizchi 's approach and my approach will be that i am attempting to fool the tensorflow library into believing it is running under non-worker thread conditions using my new found knowledge of how it works (gained by trying to fix their code). The plan is to build a faux document object and window object, complete with interfaces and values the library checks for when creating a canvas. Instead, I'll return wrapped instances of OffscreenCanvas, maybe with Proxy Traps, and catch any calls by the library to APIs I haven't stubbed out and build adaptors for those to canvas. Because TensorFlow does not ever return canvas elements to be drawn to the screen, the only overhead I'll have to worry about is sending the data from video into the worker to be processed. Because I'll be doing this with ImageBitmap, those updates will be zero-copy transferable objects (low latency). I suppose this is somewhat of a "shim" pattern and could be added to face-api.js as an adaptor or different API call if it works.

For anyone else following this path expecting that a heroic effort down this rabbit hole will maybe allow you to get this to work, a few notes you should consider:

TensorFlow (Google) is a massive project written in Typescript (Microsoft) which is made up of monolithic modules here: "dependencies": { "@tensorflow/tfjs-converter": "1.1.0", "@tensorflow/tfjs-core": "1.1.0", "@tensorflow/tfjs-data": "1.1.0", "@tensorflow/tfjs-layers": "1.1.0" } Each of those has some dependencies of their own. You'll end up having to update core, then update all the other modules the depend on core, then rebuild the whole tensorflow project with the same version of typescript, which in the case of this feature set, would be "next" or 3.5.0+, all of which aren't compatible with that version out of the box (at this time). Typescript appears to be driven primarily by features in the IE/Edge browser suite because Microsoft owns that project, and TensorFlow being Google but subject to limitations in Typescript means they A) have their own blessed version of Typescript, B) this is older than whatever the most recent release is, and C) is not as up to date with the features of the web as Google's Chrome browser team support. Maybe, eventually, once MS moves Edge to the Chromium/Blink/Whatever engine, possibly Typescript becomes one with the universe and offers support for these DOM features in sync with minimally Chrome and Edge, but ideally all major browsers. That would be awesome! But, back to the topic, the issues I ended up seeing were related to compilation errors stemmed from code that was doing type conversion like " float32ToTypedArray ", including some map functions/loops. Those were throwing errors at compile time, but unit tests ran fine. I was never able to get browserstack tests to run correctly, so I'm not sure if it really did work in the browser or not. Best of luck! @ me if you want to chat!

[jeffreytgilbert]

function isBrowser() { return typeof window === 'object' && typeof document !== 'undefined' && typeof HTMLImageElement !== 'undefined' && typeof HTMLCanvasElement !== 'undefined' && typeof HTMLVideoElement !== 'undefined' && typeof ImageData !== 'undefined'; }

This is a horrible function. If someone (me) wants to fake out a library into thinking it's in a browser, don't stifle that person by doing some oddball browser check (this is not how you detect if you're in a browser) and then be really silent and confusing when the library errors. I've been working against an error i thought was in tensorflow for hours only to realize it came from face-api.js code:

Error: getEnv - environment is not defined, check isNodejs() and isBrowser()

Tensorflow already has browser and node checks and it's own idea of environment. Why did you guys reinvent the wheel? :|

[justadudewhohacks]

I've been working against an error i thought was in tensorflow for hours only to realize it came from face-api.js code

I agree, that the error message might not be the best, but by looking at the stack trace one could have figured where the error message comes from.

The browser check is that complex, because we want to only initialize the corresponding environment of the library in case we are in a valid browser or nodejs environment to avoid errors at runtime. In any other case, it is up to the user to initialize the environment manually. All environment specifics can be monkey patched using faceapi.env.monkeyPatch.

[jeffreytgilbert]

Ok, so I'm posting this update to let everyone in this thread know that it is possible today to fool both tensorflow and face-api.js into running in a web worker and that they will run GPU accelerated, however you shouldn't get your hopes way up for perfectly jank free UX.

In my app, face detection takes approximately 60ms on a MacBook Pro (Retina, 15-inch, Mid 2015), which is only processing 640x480 stills. The stills are transferred to the worker using zero copy, so they avoid the serialize/deserialize and structured copy performance hits.

The app itself is only taking 1-2ms for any given RAF cycle, but visual jank is still occurring on Chrome when the worker thread takes longer than expected. I'm not even seeing any GC issues. The jank appears to happen while processing microtasks. I see bunches of timers being set. I'd have to look further into the face-api.js source to see if it's breaking apart workloads into chunks using 0ms setTimeout calls. If it is, those should be converted to Promises. Allowing the browser to handle batch processing stacks of timeouts will definitely result in slower performance if that's what's happening. Timeouts can take 2-4ms to resolve and Promises are almost immediate. I believe the details of how promise scheduling is done is still on a per browser basis, but if you're in this thread, you're today only interested in the ones that support OffscreenCanvas, and that's Chrome. Chrome handles them async.

Here's the admittedly over engineered code for creating a worker environment that tensorflow and face-api.js will run in:

Parent `

    var screenCopy = {};
    for(let key in screen){
        screenCopy[key] = +screen[key];
    }
    screenCopy.orientation = {};
    for(let key in screen.orientation){
        if (typeof screen.orientation[key] !== 'function') {
            screenCopy.orientation[key] = screen.orientation[key];
        }
    }

    var visualViewportCopy = {};
    if (typeof window['visualViewport'] !== 'undefined') {
        for(let key in visualViewport){
            if(typeof visualViewport[key] !== 'function') {
                visualViewportCopy[key] = +visualViewport[key];
            }
        }
    }

    var styleMediaCopy = {};
    if (typeof window['styleMedia'] !== 'undefined') {
        for(let key in styleMedia){
            if(typeof styleMedia[key] !== 'function') {
                styleMediaCopy[key] = styleMedia[key];
            }
        }
    }

    let fakeWindow = {};
    Object.getOwnPropertyNames(window).forEach(name => {
        try {
            if (typeof window[name] !== 'function'){
                if (typeof window[name] !== 'object' && 
                    name !== 'undefined' && 
                    name !== 'NaN' && 
                    name !== 'Infinity' && 
                    name !== 'event' && 
                    name !== 'name' 
                ) {
                    fakeWindow[name] = window[name];
                } else if (name === 'visualViewport') {
                    console.log('want this?', name, JSON.parse(JSON.stringify(window[name])));
                } else if (name === 'styleMedia') {
                    console.log('want this?', name, JSON.parse(JSON.stringify(window[name])));
                }
            }
        } catch (ex){
            console.log('Access denied for a window property');
        }
    });

    fakeWindow.screen = screenCopy;
    fakeWindow.visualViewport = visualViewportCopy;
    fakeWindow.styleMedia = styleMediaCopy;
    console.log(fakeWindow);

    let fakeDocument = {};
    for(let name in document){
        try {
            if(name === 'all') {
                // o_O
            } else if (typeof document[name] !== 'function' && typeof document[name] !== 'object') {
                    fakeDocument[name] = document[name];
            } else if (typeof document[name] === 'object') {
                fakeDocument[name] = null;
            } else if(typeof document[name] === 'function') {
                fakeDocument[name] = { type:'*function*', name: document[name].name };
            }
        } catch (ex){
            console.log('Access denied for a window property');
        }
    }

`

Worker `

Canvas = HTMLCanvasElement = OffscreenCanvas; HTMLCanvasElement.name = 'HTMLCanvasElement'; Canvas.name = 'Canvas';

function HTMLImageElement(){} function HTMLVideoElement(){}

Image = HTMLImageElement; Video = HTMLVideoElement;

// Canvas.prototype = Object.create(OffscreenCanvas.prototype);

function Storage () { let _data = {}; this.clear = function(){ return _data = {}; }; this.getItem = function(id){ return _data.hasOwnProperty(id) ? _data[id] : undefined; }; this.removeItem = function(id){ return delete _data[id]; }; this.setItem = function(id, val){ return _data[id] = String(val); }; } class Document extends EventTarget {}

let window, document = new Document();

        // do terrible things to the worker's global namespace to fool tensorflow
        for (let key in event.data.fakeWindow) {
            if (!self[key]) {
                self[key] = event.data.fakeWindow[key];
            } 
        }
        window = Window = self;
        localStorage = new Storage();
        console.log('*faked* Window object for the worker', window);

        for (let key in event.data.fakeDocument) {
            if (document[key]) { continue; }

            let d = event.data.fakeDocument[key];
            // request to create a fake function (instead of doing a proxy trap, fake better)
            if (d && d.type && d.type === '*function*') {
                document[key] = function(){ console.log('FAKE instance', key, 'type', document[key].name, '(',document[key].arguments,')'); };
                document[key].name = d.name;
            } else {
                document[key] = d;
            }
        }
        console.log('*faked* Document object for the worker', document);

        function createElement(element) {
            // console.log('FAKE ELELEMT instance', createElement, 'type', createElement, '(', createElement.arguments, ')');
            switch(element) {
                case 'canvas':
                    // console.log('creating canvas');
                    let canvas = new Canvas(1,1);
                    canvas.localName = 'canvas';
                    canvas.nodeName = 'CANVAS';
                    canvas.tagName = 'CANVAS';
                    canvas.nodeType = 1;
                    canvas.innerHTML = '';
                    canvas.remove = () => { console.log('nope'); };
                    // console.log('returning canvas', canvas);
                    return canvas;
                default:
                    console.log('arg', element);
                    break;
            }
        }

        document.createElement = createElement;
        document.location = self.location;
        console.log('*faked* Document object for the worker', document);

`

[jeffreytgilbert]

Here's what I'm seeing btw. I'm going to try your suggestion of looking at using a different model that might process faster.

[jeffreytgilbert]

Check this out. This is what I'm talking about when I'm making this correlation. When timers are used in bulk, they appear to mess up the process scheduling by spamming the event loop. Promises don't appear to have the same problem. Chrome bundles them up nicely and still has the ability to handle requestAnimationFrame requests. I'd like to see if there's a way in face-api.js to fix the workload splitting so it doesn't rely on setTimeout

[justadudewhohacks]

I'd like to see if there's a way in face-api.js to fix the workload splitting so it doesn't rely on setTimeout

Hmm, actually there are no calls to setTimeout, tf.nextFrame or requestAnimationFrame in face-api.js. Could it be, that the async behaviour you are encountering here is due to downloading data from the GPU via tf.data()?

[jeffreytgilbert]

ok, possibly disproved the timer spam theory. I'm now pointing to the GPU work. While I was overwriting everything sacred (window, document, etc) I rewrote the setTimeout function so it uses promises and request animation frame for 0 ms setTimeouts, and falls back to setInterval for actual timers. It worked exactly how I anticipated it would, except that the jank is still present and the only thing left to point a finger at is the GPU load that's 2 frames long. 👎

So, for everyone watching, probably keep your GPU load in mind. It can block things just like anything else.

[jeffreytgilbert]

Timeout replacement code

// More really bad practices to fix closed libraries. Here we overload setTimeout to replace it with a flawed promise implementation which sometimes cant be canceled.

let callStackCount = 0; const maxiumCallStackSize = 750; // chrome specific 10402, of 774 in my tests

setTimeout = function (timerHandler, timeout) { let args = Array.prototype.slice.call(arguments); args = args.length <3 ? [] : args.slice(2, args.length); if (timeout === 0) { if (callStackCount < maxiumCallStackSize) { var cancelator = {cancelable: false }; callStackCount++; new Promise(resolve=>{ resolve(timerHandler.apply(self, args)); }); return cancelator; } else { requestAnimationFrame(()=>{ timerHandler.apply(self, args); }); callStackCount = 0; return; } } const i = setInterval(()=>{ clearInterval(i); timerHandler.apply(self, args); }, timeout); return i; };

clearTimeout = (id)=>{ console.log(id); if(id && id.cancelable === false) { console.error('woops. cant cancel a 0ms timeout anymore! already ran it'); } else { clearInterval(id);} };

// var x = setTimeout((x,y,z)=>{console.log(x,y,z);}, 0, 'hello', 'im', 'cassius'); // var y = setTimeout((x,y,z)=>{console.log(x,y,z);}, 1000, 'hello', 'im', 'cassius'); // clearTimeout(x); // clearTimeout(y);

[hyakki]

Is there any other "cleaner" way to do this ?

@justadudewhohacks you mentioned the faceapi.env.monkeyPatch but how does it work exactly ?

I mean, lets say I have a main.js that only do this:

const worker = new Worker('worker.js');

worker.postMessage('foo');

and a worker where I want to be able to do this:

import * as faceapi from 'face-api.js';

faceapi.loadFaceExpressionMode('assets/models/');

onmessage = function(event) {
  console.log(event);
}

where and how should I use the faceapi.env.monkeyPatch ?

The error raised atm is the following:Uncaught (in promise) Error: getEnv - environment is not defined, check isNodejs() and isBrowser().

[justadudewhohacks]

@maximeparisse you would monkey patch environment specific after importing the package. In the nodejs examples we monkey patch Canvas, Image and ImageData for example, as shown here.

Refer to the Environment type to see what can be overridden.

[hyakki]

@justadudewhohacks : Thank you for your reply, i will give a shot and give my feedbacks here in case that can help others.

[hyakki]

@justadudewhohacks : I've tried without success to do that in a web worker. I understand how you patched the env spec in nodejs but i can't see how i can reproduce it for a web worker.

[jeffreytgilbert]

@maximeparisse face-api.js only looks for those native methods as a node server vs browser check, per my example above. There are also tfjs detections. You have to set those values inside the worker before loading the libraries in order to fool the libraries into believing they're in a browser. If they fall into the node detection block, they will fail that check too, then bail out to a null result rather than a default (browser). A good patch to apply to face-api.js would be to add worker cases and change the detection to if/elseif/elseif/else style blocks so there is always a default case and more reasonable fallbacks. This is doable, but the trick is in supporting workers for browsers other than Chrome or Firefox with the flag set to enable OffscreenCanvas.

[ivanbacher]

Anyone manage to get this working?

[jeffreytgilbert]

Yes. Turned out the integrated gpu was the biggest insurmountable bottleneck to avoid blocking the rendering pipeline. I’d be willing to revisit this once tensorflow and this lib have been updated to allow for offscreencanvas support, which is necessary to avoid excessive monkey patching and environmental fake outs to the two libs.

[jeffreytgilbert]

Also: https://caniuse.com/#feat=offscreencanvas

[josiahbryan]

@jeffreytgilbert It appears TensorFlow.js now supports Offscreen Canvas... At least according to this article: https://medium.com/@wl1508/webworker-in-tensorflowjs-49a306ed60aa - does that jive with what you're seeing? Should face-api/tfjs "just work" in WebWorkers now...?

[josiahbryan]

I can sadly confirm that face-api does not Just Work, even with monkeyPatch.

When I do the following in my worker:

import * as faceapi from 'face-api.js';

faceapi.env.monkeyPatch({ Canvas: OffscreenCanvas })

I get:

Uncaught Error: monkeyPatch - environment is not defined, check isNodejs() and isBrowser()
    at Object.monkeyPatch (index.ts:38)

I've checked the isBrowser module, and I've done a TON of monkey patching of my own BEFORE calling monekyPatch(), and got the following checks to pass in my own code:

// isBrowserCheck is true in my tests
const isBrowserCheck = typeof window === 'object'
&& typeof document !== 'undefined'
&& typeof HTMLImageElement !== 'undefined'
&& typeof HTMLCanvasElement !== 'undefined'
&& typeof HTMLVideoElement !== 'undefined'
&& typeof ImageData !== 'undefined'
&& typeof CanvasRenderingContext2D !== 'undefined';

My own monkey patching is based on @jeffreytgilbert 's example above, with a few edits to make it compile, and I added CanvasRenderingContext2D = OffscreenCanvasRenderingContext2D;.

Bottom line: faceapi.env.monkeyPatch does not even try to monkey patch because of the error above.

Anyone have any suggestions on how to get this to even work? GPU or no GPU, I just want to try to get it to work. (Chrome 79 on a brand new MacBook Pro 15", so yes, OffscreenCanvas is supported.)

[josiahbryan]

Update: Got it working.

How? Use this gist: https://gist.github.com/josiahbryan/770ca1a9d72f1b35c13219ba84dc0495

Import it into your worker. If you have a bundler setup for your worker, just do (assuming you put it in your utils/ folder):

import './utils/faceEnvWorkerPatch';

Don't need to call faceapi's monkeyPatch if you use that.

Fair warning: That gist is NOT pretty. It is a conglomeration of hacks and workarounds and whatever else. But it works. Face detection is working for me now in a web worker.

Ideally, face-api would support a WebWorker WITHOUT having to do that horrendous hack of a monkey patch I just uploaded, but, yeah. At least this works now.

[wodnjs6512]

I found my own way of monkey patching. Pretty simple but only supports OffscreenCanvas,

faceapi.env.setEnv(faceapi.env.createNodejsEnv());

faceapi.env.monkeyPatch({
    Canvas: OffscreenCanvas,
    createCanvasElement: () => {
        return new OffscreenCanvas(480, 270);
    },
});

No need to import canvas, supports OffscreenCanvas rigidly, and seems this is the easiest valid way.

[ShaharGigi]

Hi all, I'm trying to get the faceapi work on an HTMLImageElements that I send from the DOM to the web worker. I can't use the offscreencanvas as I use its context in the main thread for rendering. Any advice? I use @josiahbryan monkeypatch which helps with making Image defined in the worker but then I get: Uncaught (in promise) Error: getEnv - environment is not defined, check isNodejs() and isBrowser()

[josiahbryan]

Look at the "transferControlToOffscreen" method of OffscreenCanvas ...

On Thu, Feb 20, 2020 at 1:51 PM ShaharGigi notifications@github.com wrote:

Hi all, I'm trying to get the faceapi work on HTMLImageElements that I send from the DOM to the web worker. I can't use the offscreencanvas as I use its context in the main thread for rendering. Any advice?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/justadudewhohacks/face-api.js/issues/47?email_source=notifications&email_token=ABEZELBZVQH5IQJDNWMSM2LRDZVFXA5CNFSM4FJTKRRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMNSQHA#issuecomment-588982300, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEZELHEPFZH2L4LWVMBDITRDZVFXANCNFSM4FJTKRRA .

-- Josiah Bryan 765-215-0511 www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig josiahbryan@gmail.com

[ivanbacher]

How would you monkeyPatch the HTMLImageElement in webworkers?

In the Node.js example we are using an external library.

// implements nodejs wrappers for HTMLCanvasElement, HTMLImageElement, ImageData
import * as canvas from 'canvas';

// patch nodejs environment, we need to provide an implementation of
// HTMLCanvasElement and HTMLImageElement, additionally an implementation
// of ImageData is required, in case you want to use the MTCNN
const { Canvas, Image, ImageData } = canvas;
faceapi.env.monkeyPatch({ Canvas, Image, ImageData});

Is there anything similar that can be used for webworkers.

E.g. these are the example steps that are called in the web worker

1) load the image using fetch -> returns a blob

function loadImage(url) {
    let p = new Promise( (resolve) => {

      fetch(url)
      .then( (response) => {
        if (!response.ok) {
          throw Error(response.statusText);
        }
        return response.blob();
      })
      .then( (blob) => {
        resolve(blob);
      })
    });

    return p;
  }

2) Create Image from blob

???

3) Use face-api to detect face/faces

let detections = await faceapi
          .detectSingleFace(image) // not sure how to get image in webworker
          .withFaceLandmarks()
          .withFaceDescriptor();

[josiahbryan]

@ivanbacher I know I can use canvas in WebWorker, and faceapi can convert ImageData to a canvas for you in WebWorker - so what about rendering your Image to a Canvas in the main thread, then using the canvas getImageData function to grab the bytes, and then send it to the WebWorker in a message (transfer the buffer), then make ImageData out of the buffer, and use that ImageData with faceapi? I know that route does work, I do it even today in my own code (though the canvas gets video in the main thread instead of an image, but you can easily render an image to the main thread canvas and it will be the same.)

Let me know if you have questions on that route...

[muebau]

@ivanbacher do you have an example for that scenario (main->image->worker->face-api->json->main). I got stuck with face-api and WebWorker and there is simply no error and nothing I could find. I think I mess up some part I simply don't understand in the first place.

[ivanbacher]

@josiahbryan thanks for the pointer. Makes a lot of sense.

@muebau sure. Look at the code below. when implementing you have to take a couple of things into consideration: control flow, face-api setup, etc. But I think it gets the point across. Hope this helps

Main Thread

function getImage() {
    let p = new Promise( (resolve) => {
        let img = new Image();
        img.onload = ()=>{ resolve(img); }
        img.src = url
   });
  return p;
}

getImage().then( (img) => {
    let canvas = new OffscreenCanvas(img.width, img.height); //can also be normal canvas
    let context = canvas.getContext('2d');
    context.drawImage(img, 0, 0, img.width, img.height);

   let imgData = context.getImageData(0, 0, img.width, img.height);

    worker.postMessage( { 
        type:'send-img-data',
        w: img.width, 
        h: img.height, 
        buffer: imgData.data.buffer 
    }, [imgData.data.buffer] );
});

In Worker

onmessage = (e) => {
    let buffer = e.data.buffer;
    let w = e.data.w; 
    let h = e.data.h;

    let canvas = new OffscreenCanvas(w, h);
    let context = canvas.getContext('2d');
    let imageData = context.createImageData(w, h);

    let arr = new Uint8ClampedArray(data.buffer);
    imageData.data.set(arr);

    context.putImageData(imageData, 0, 0);

     //now the image should be drawn to the offscreencanvas

    //you can use offscreen canvas as input to faceapi methods e.g.

    let detections = await faceapi.detectSingleFace(canvas)
            .withFaceLandmarks().withFaceDescriptor();
}

[ivanbacher]

@aendrew could be, I haven't actually tried it with TinyFaceDetector

[aendra-rininsland]

@ivanbacher Drat, you were too quick! 😅 I deleted my comment because I realised I forgot to monkeypatch. Your workflow seems to work fine. 👍

[ayadima]

Update: Got it working.

How? Use this gist: https://gist.github.com/josiahbryan/770ca1a9d72f1b35c13219ba84dc0495

Import it into your worker. If you have a bundler setup for your worker, just do (assuming you put it in your utils/ folder):

import './utils/faceEnvWorkerPatch';

Don't need to call faceapi's monkeyPatch if you use that.

Fair warning: That gist is NOT pretty. It is a conglomeration of hacks and workarounds and whatever else. But it works. Face detection is working for me now in a web worker.

Ideally, face-api would support a WebWorker WITHOUT having to do that horrendous hack of a monkey patch I just uploaded, but, yeah. At least this works now.

@josiahbryan I used your patch and it does work locally but when i build the project, the imported patch is not accepted and I get "Uncaught ReferenceError: HTMLCanvasElement is not defined" I am using React.JS with react-rewired and worker-loader. Can you or anyone else help me with this ?

[josiahbryan]

Sounds like your prod build is running extra linter checks that your local dev build is not doing.

Without being able to recreate your compile error here (as it works fine for me right now...), I would start by trying to change lines 4 in the faceEnvWorkerPatch FROM:

Canvas = HTMLCanvasElement = OffscreenCanvas;

TO

self.Canvas = self.HTMLCanvasElement = OffscreenCanvas;

I'm guessing that's the line that it doesn't like, but you would have to say for sure since there was no line number in your message.

The idea here is that yeah, HTMLCanvasElement of COURSE is not defined - duh. We're trying to define it lol! Since the global context object in the worker is self, we (theoretically) can avoid your linter error by telling the compiler we're defining a new property on the self object instead of relying on the linter to realize that's what we're trying to do.

Anyway, yeah, see if that works. If not, looks at the line # where it's complaining about being undefined and see if you can fix it with those hints above, or ping back here with more details, or maybe somebody else knows better haha. Good luck!! 😄

[ayadima]

Sounds like your prod build is running extra linter checks that your local dev build is not doing.

Without being able to recreate your compile error here (as it works fine for me right now...), I would start by trying to change lines 4 in the faceEnvWorkerPatch FROM:

Canvas = HTMLCanvasElement = OffscreenCanvas;

TO

self.Canvas = self.HTMLCanvasElement = OffscreenCanvas;

I'm guessing that's the line that it doesn't like, but you would have to say for sure since there was no line number in your message.

The idea here is that yeah, HTMLCanvasElement of COURSE is not defined - duh. We're trying to define it lol! Since the global context object in the worker is self, we (theoretically) can avoid your linter error by telling the compiler we're defining a new property on the self object instead of relying on the linter to realize that's what we're trying to do.

Anyway, yeah, see if that works. If not, looks at the line # where it's complaining about being undefined and see if you can fix it with those hints above, or ping back here with more details, or maybe somebody else knows better haha. Good luck!! 😄

Thank you for your answer ! it worked with adding "self." ! and yes it was a linting problem when building !

[stephenjason89]

I found my own way of monkey patching. Pretty simple but only supports OffscreenCanvas,

faceapi.env.setEnv(faceapi.env.createNodejsEnv());

faceapi.env.monkeyPatch({
    Canvas: OffscreenCanvas,
    createCanvasElement: () => {
        return new OffscreenCanvas(480, 270);
    },
});

No need to import canvas, supports OffscreenCanvas rigidly, and seems this is the easiest valid way.

Thank you works flawlessly

Main Thread

function getImage() {
    let p = new Promise( (resolve) => {
        let img = new Image();
        img.onload = ()=>{ resolve(img); }
        img.src = url
   });
  return p;
}

getImage().then( (img) => {
    let canvas = new OffscreenCanvas(img.width, img.height); //can also be normal canvas
    let context = canvas.getContext('2d');
    context.drawImage(img, 0, 0, img.width, img.height);

   let imgData = context.getImageData(0, 0, img.width, img.height);

    worker.postMessage( { 
        type:'send-img-data',
        w: img.width, 
        h: img.height, 
        buffer: imgData.data.buffer 
    }, [imgData.data.buffer] );
});

In Worker

onmessage = (e) => {
    let buffer = e.data.buffer;
    let w = e.data.w; 
    let h = e.data.h;

    let canvas = new OffscreenCanvas(w, h);
    let context = canvas.getContext('2d');
    let imageData = context.createImageData(w, h);

    let arr = new Uint8ClampedArray(data.buffer);
    imageData.data.set(arr);

    context.putImageData(imageData, 0, 0);

     //now the image should be drawn to the offscreencanvas

    //you can use offscreen canvas as input to faceapi methods e.g.

    let detections = await faceapi.detectSingleFace(canvas)
            .withFaceLandmarks().withFaceDescriptor();
}

Thanks for this as well, I do have a question though.

1. Is getting the imgData buffer from the main thread then sending it to the worker to do the detection and draw the results to an offscreen canvas then getting the ImgData of the offscreen canvas to postmessage to the main thread, faster and less resource intensive?

[josiahbryan]

Hey about your questions on displaying it on a canvas, you actually can just render to a canvas from INSIDE the web worker and it will AUTOMATICALLY update the canvas outside the webworker.

How?

Well, in my code, I passed in the canvas from the main thread like this:

const canvasFromMainThread = detectionResultsCanvas.transferControlToOffscreen();

myWebWorker.postMessage({
    type: "setCanvasFromMainThread",
    canvasFromMainThread,
}, [ canvasFromMainThread ]);

Notice the array at the end of the postMessage() call - that transfers the the canvas to the web worker thread instead of copying it. (See MDN on the topic of postMessage and transfer lists: https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)

In my web worker, I receive the message and store canvasFromMainThread for later.

Then, in my web worker, after getting the detection results from the neural net, I did the usual thing:

const ctx = canvasFromMainThread.getContext('2d');
// Have to clear it because everytime before we render results
ctx.clearRect(0, 0, canvasFromMainThread.width, canvasFromMainThread.height);

faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));

That's all that's needed - the canvas in the main thread will automatically display the results drawn. You don't have to send the detection results or the canvas back to the main thread. It all works magically.

One tip: The detectionResultsCanvas that I pass IN to the web worker will NOT have the video/image on it that the web worker is using - it is only for OUTPUT of the results.

So, in my main thread, what I really have is something like this:

• CanvasContainer
  • Canvas 1 - LiveVideoCanvas
  • Canvas 2 - OverlayCanvas

I use simple CSS to make the LiveVideoCanvas be position: absolute; top: 0; left: 0; z-index: 0, and then the OverlayCanvas is position: absolute; top: 0; left: 0; z-index: 1

Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.

Make sense? Feel free to ping with questions!

[stephenjason89]

Hey about your questions on displaying it on a canvas, you actually can just render to a canvas from INSIDE the web worker and it will AUTOMATICALLY update the canvas outside the webworker.

How?

Well, in my code, I passed in the canvas from the main thread like this:

const canvasFromMainThread = detectionResultsCanvas.transferControlToOffscreen();

myWebWorker.postMessage({
    type: "setCanvasFromMainThread",
    canvasFromMainThread,
}, [ canvasFromMainThread ]);

Notice the array at the end of the postMessage() call - that transfers the the canvas to the web worker thread instead of copying it. (See MDN on the topic of postMessage and transfer lists: https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)

In my web worker, I receive the message and store canvasFromMainThread for later.

Then, in my web worker, after getting the detection results from the neural net, I did the usual thing:

const ctx = canvasFromMainThread.getContext('2d');
// Have to clear it because everytime before we render results
ctx.clearRect(0, 0, canvasFromMainThread.width, canvasFromMainThread.height);

faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));

That's all that's needed - the canvas in the main thread will automatically display the results drawn. You don't have to send the detection results or the canvas back to the main thread. It all works magically.

One tip: The detectionResultsCanvas that I pass IN to the web worker will NOT have the video/image on it that the web worker is using - it is only for OUTPUT of the results.

So, in my main thread, what I really have is something like this:

• CanvasContainer

  • Canvas 1 - LiveVideoCanvas
  • Canvas 2 - OverlayCanvas

I use simple CSS to make the LiveVideoCanvas be position: absolute; top: 0; left: 0; z-index: 0, and then the OverlayCanvas is position: absolute; top: 0; left: 0; z-index: 1

Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.

Make sense? Feel free to ping with questions!

Thank you for the detailed reply! I just finished doing everything before i saw your reply and ended up exactly as what you said.

My last question though would be, how did you implement passing the live feed images to your webworker?

What i did was create an ImageBitmap from the live feed and pass it to the worker as a transferable. on the worker, i ended up creating a new offscreen canvas and using .drawImage(imageBitmap,0 ,0)

Would like to know if there's a better way :)

Thank you!

[josiahbryan]

Not really. I ended up passing individual frames into the worker at about 12fps (I set a timeout).

The core of it looks like in the main thread:

// Use this to grab video frame for sending to web worker
const faceCaptureCtx = this.faceCaptureCanvas.getContext('2d');
faceCaptureCtx.drawImage(this.localVideo, 0, 0);
const imgData = faceCaptureCtx.getImageData(0, 0, this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);

const { height, width, data } = imgData;
// Transfer the buffer via a transferlist
// https://stackoverflow.com/questions/41497124/allowable-format-for-webworker-data-transfer
ProtoCamCore.worker.postMessage({
    type: "frame",
    height,
    width,
    data,
}, [ data.buffer ]);

And in the web worker:

// props is the message from the main thread
const imgData = new ImageData(
   new Uint8ClampedArray(props.data),
    props.width,
    props.height
);

// Create a canvas from our rgbaBuffer
const img = faceapi.createCanvasFromMedia(imgData);

const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

[stephenjason89]

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

Thank you! I don't really know if this will improve anything but based on this thread https://stackoverflow.com/questions/60031536/difference-between-imagebitmap-and-imagedata

imageBitmap is stored directly to the GPU and imgData in the CPU so this might help a bit in terms of performance

let video = document.getElementById('video');
let offlineCanvas = new OffscreenCanvas(video.width, video.height);
let context = offlineCanvas.getContext('2d');

context.drawImage(video, 0, 0, video.width, video.height);

let bmp = offlineCanvas.transferToImageBitmap();
worker.postMessage({
            image: bmp
        }, [bmp]);

then you can get width and height from worker by the image.width and image.height props like this and just draw the image to a canvas, either to a new canvas or an existing offscreencanvas.

canvas = new OffscreenCanvas(e.data.image.width, e.data.image.height);
    context = canvas.getContext('2d');
    context.drawImage(e.data.image, 0, 0);

let me know if it improves the performance.

I don't haven't tried other benchmarking tools. I just use chrome dev tools light house but can't get an accurate measurement.

on the link, they used benchmark.js to measure it.

comparing ctx.putImageData(imageData, 0, 0); with ctx.drawImage(imageBitmap, 0, 0);

results

ImageData x 13,473 ops/sec ±11.49% (74 runs sampled)
ImageBitmap x 271,153 ops/sec ±4.62% (77 runs sampled)

Thank you

One offtopic question though, using faceMatcher.findBestMatch(singleResult.descriptor) or the euclideanDistance, what distance is acceptable for you?

I find that my wife is sometimes recognized as my daughter with distance > 0.4

[josiahbryan]

Appreciate that!

I don't have that project running right now so I can't really measure performance either - I'm knee deep in another code base on some stuff haha!

But, when I do return to that project, I'll try that out! Good find!

On Fri, Apr 24, 2020 at 7:07 AM stephenjason89 notifications@github.com wrote:

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

Thank you! I don't really know if this will improve anything but based on this thread

https://stackoverflow.com/questions/60031536/difference-between-imagebitmap-and-imagedata

imageBitmap is stored directly to the GPU and imgData in the CPU so this might help a bit in terms of performance

let bmp = offlineCanvas.transferToImageBitmap(); worker.postMessage({ image: bmp }, [bmp]);

then you can get width and height from worker by the image.width and image.height props.

let me know if it improves the performance.

I don't know how to measure it. I just use chrome dev tools light house but can't get an accurate measurement.

Thank you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-618790216, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEZELEZX47LU733GFTQNOTROEGB3ANCNFSM4FJTKRRA .

-- Josiah Bryan +972-050-794-0566 www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig josiahbryan@gmail.com

[flatsiedatsie]

Just a quick thank you to all who shared code in this thread. I'm a beginner, but managed to get everything running in a webworker. Now the interface doesn't completely freeze, which is great. It does feel like it runs a but slower - especially the initial run - but it's a worthwhile tradeoff.

[patriciogutierrez]

Just a quick thank you to all who shared code in this thread. I'm a beginner, but managed to get everything running in a webworker. Now the interface doesn't completely freeze, which is great. It does feel like it runs a but slower - especially the initial run - but it's a worthwhile tradeoff.

can you share the code? I can't load the models. Nevermind. It's working. Thank you all

[nevillekatila]

Calling the following line recursively causes a memory leak and eventually results in a crash:

const imgData = faceCaptureCtx.getImageData(0, 0, this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);

It seems that there is a known memory leak issue in calling getImageData on the context object of a canvas repeatedly in short intervals.

Has anyone experienced this? Any suggestions?

On Fri, Apr 24, 2020 at 3:20 AM Josiah Bryan notifications@github.com wrote:

Not really. I ended up passing individual frames into the worker at about 12fps (I set a timeout).

The core of it looks like in the main thread:

// Use this to grab video frame for sending to web worker const faceCaptureCtx = this.faceCaptureCanvas.getContext('2d'); faceCaptureCtx.drawImage(this.localVideo, 0, 0); const imgData = faceCaptureCtx.getImageData(0, 0, this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);

  const { height, width, data } = imgData;
  // Transfer the buffer via a transferlist
  // https://stackoverflow.com/questions/41497124/allowable-format-for-webworker-data-transfer
  ProtoCamCore.worker.postMessage({
      type: "frame",
      height,
      width,
      data,
  }, [ data.buffer ]);

And in the web worker:

// props is the message from the main thread const imgData = new ImageData( new Uint8ClampedArray(props.data), props.width, props.height );

// Create a canvas from our rgbaBuffer const img = faceapi.createCanvasFromMedia(imgData);

const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-618690317, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFZFNZRVOOQKRE6LF7LOKZTROCZ3XANCNFSM4FJTKRRA .