Using FaceAPI in worker with "Allow WebGL in Web Workers" enabled in Safari

[SirVizzy]

I understand this library wasn't meant to be ran in a worker. However, after using the monkey patch is actually works like a charm in most cases.

Issue Description By default, on my iPhone XR the feature flag "Allow WebGL in Web Workers" is disabled. Since WebGL is not enabled FaceAPI defaults to wasm. Which is perfect. However, enabling this feature breaks FaceAPI. I am getting the following message from the worker;

{
    "error": "Type error",
    "trace": "texSubImage2D@[native code]\nde@\nyA@\nuploadPixelDataToTexture@\nXae@\no@\n@\nscopedRun@\nrunKernelFunc@\nr@\nmap@[native code]\n@\n@\nscopedRun@\n@\n@\nscopedRun@\n@"
}

Steps to Reproduce I used the following patch to make it work in workers;

const setup = () => {

    // Environment must be patched in order to work with OffscreenCanvas.
    // Source: https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-587379255
    patchEnvironment();

    return Promise.all([
        waitForBackend(),
        loadModels()
    ]);
}

const waitForBackend = () => {
    tf.setWasmPaths(`${self.location.origin}/assets/models/backend/`);
    return tf.ready();
};

const patchEnvironment = () => {
    env.setEnv(env.createNodejsEnv());
    env.monkeyPatch({
        Canvas: OffscreenCanvas,
        createCanvasElement: createOffscreenCanvas
    });
};

const loadModels = () => {
    return nets.ssdMobilenetv1.loadFromUri(`${self.location.origin}/assets/models/face-detection`);
};

Using this patch enable "Allow WebGL in Web Workers" in the safari advanced settings. After, run the face detection and see whether it breaks.

Expected Behavior I expect the backend to be switched to WebGL (I tested this with an Android phone, and it worked). Or, if not working, fallback to either wasm or cpu.

Environment

• Module version?
• • 1.7.13
• Built-in demo or custom code?
• • Custom
• Type of module used (e.g. js, esm, esm-nobundle)?
• • JS
• Browser or NodeJS and version (e.g. NodeJS 14.15 or Chrome 89)?
• • Browser, patched like node in a worker (Safari + Chrome)
• OS and Hardware platform (e.g. Windows 10, Ubuntu Linux on x64, Android 10)?
• • iPhoneXR
• Packager (if any) (e.g, webpack, rollup, parcel, esbuild, etc.)?
• • Webpack

• ---

[vladmandic]

I understand this library wasn't meant to be ran in a worker. However, after using the monkey patch is actually works like a charm in most cases.

I'm glad it does, but its still unsupported since FaceAPI internally relies on DOM elements a lot.

I expect the backend to be switched to WebGL (I tested this with an Android phone, and it worked). Or, if not working, fallback to either wasm or cpu.

Fallback does happen if backend is unavailable. If its available, but non-functional, its really up to user to setup rules. You could detect platform and set appropriate backend yourself.

In either case, this is not something that will be worked on.

[SirVizzy]

I'm glad it does, but its still unsupported since FaceAPI internally relies on DOM elements a lot.

It really does work well. I'm quite happy with the <50ms detection time for such model.

In either case, this is not something that will be worked on.

Thought so. Do you have any hunch on why this might go wrong given the error?

Fallback does happen if backend is unavailable. If its available, but non-functional, its really up to user to setup rules. You could detect platform and set appropriate backend yourself.

Yeah. I'm planning on doing this. However I want to rely on the fallback mechanism as much as possible. I'm unsure what to check for as it does work for another iOS device.

[vladmandic]

I'm unsure what to check for as it does work for another iOS device.

neither do i :(