remove numeric new tensorflow model "face-landmarks-detection"

[koll93]

This PR replaced numeric with mathjs to fix the CSP error as mentioned in #211. This is a new version https://github.com/brownhci/WebGazer/pull/226 with actual model "@tensorflow-models/face-landmarks-detection". This works in google chrome extension with manifest v3. Note the webpack setting optimization.minimize: true, CSP will throw an error if the file is large.

[koll93]

Can anyone confirm if it works?

[jeffhuang]

@steviebaa I saw you had expressed some enthusiasm for this. Could you test? I don't know much about chrome extensions, so it might be nice for someone who works with them to try this out.

[jeffhuang]

@Skylion007 do you remember how you checked the performance of this change last time?

[webzep]

sure. ill check it out. @koll93, would you mind cleaning up the formatting? It looks like your formator removed/introduced a bunch of whitespace in all the files bloating the PR.

[koll93]

@steviebaa okey)

[webzep]

@jeffhuang is there a formatting config file for this project, like a .prettierrc? It would benefit a lot of contributors.

[webzep]

@koll93 How are you using WebGazer in chrome MV3? Are you trying to use WebGazer in the Service Worker? If that's the case I think the only way to feed the media stream is via a continuous postMessage from the UI that has access to the camera.

[koll93]

@steviebaa I use contentScript create iframe import webgazer. Iframe z-index: max, width and heigth 100%. pointer-events: none or add webgazer in contentScript.

[koll93]

@steviebaa fix space.

[webzep]

Nice one Nikolai. I don't have the time to set up an mv3 project and test it at the moment. I'll try to get back around to it as soon as possible. Might be next weekend.

[lukaas33]

Hi,

I am using webgazer in an Chrome extension and also had problems with the evals of the numeric library. I found your pull request and tested it myself.

Steps executed:

1. git clone https://github.com/koll93/WebGazer.git
2. cd WebGazer
3. git checkout refactor-remove-numeric-landmarks
4. npm install
5. npm run build

After this, I moved the webgazer.min.js file from dist to my extension folder and included it like this:

  "content_scripts": [
    {
      "js": ["/assets/scripts/libraries/webgazer.min.js"],
      "matches": [
        "https://*/*"
      ]
    }

When importing the extension this doesn't fail on eval statements as it did before.

However, when opening the webpage https://developer.chrome.com/docs/extensions/mv2/content_scripts/ it throws an error:

[Report Only] Refused to compile or instantiate WebAssembly module because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' https://www.google-analytics.com https://cdnjs.cloudflare.com https://www.gstatic.com https://www.google.com https://*.firebaseio.com".

I am new at this, so I may have done something wrong. Or is there still an 'unsafe' dependency within the WebGazer library?

[koll93]

Hi. I will make an example on the weekend

[koll93]

check out the demo, you might want to use dynamic imports. Now the webgazer is loading as a second content script. https://github.com/koll93/extension-manifest3-webgazer

[lukaas33]

I tested the repo at https://github.com/koll93/extension-manifest3-webgazer by cloning it and installing it locally in my chrome.

The extension throws the following error which I think isn't relevant for the webgazer library. It also didn't interfere with camera access so I think we can ignore it. Uncaught (in promise) NotAllowedError: Permission denied

The error of my above comment is still being thrown. However as far as I can tell so far it doesn't interfere with Webgazer running.

[Report Only] Refused to compile or instantiate WebAssembly module because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' https://www.google-analytics.com/ https://cdnjs.cloudflare.com/ https://www.gstatic.com/ https://www.google.com/ https://*.firebaseio.com".

I am curious if all functions of webgazer work in Chrome extensions now and if it works on all websites. What exactly is the purpose of the webassembly code? Could it be left out or will this break some functionality?

So is there more research needed before this PR can be merged?

I would be happy to help if I needed to test anything, since I will be actively working on an extension with webgazer for a while.

[koll93]

Strangely, this assembly does not use wasm. Can anyone test it too?

[jeffhuang]

This is a pretty substantial change, and an important one. I'd love a second pair of eyes on it. We'll try to get this committed for the next version release.

[jeffhuang]

haven't forgotten about this, but would love a second pair of eyes to review

[koll93]

@jeffhuang Hi do you have time to watch?)

[jeffhuang]

I would love it if one other person can review this, especially someone with more of an interest in removing numeric than me.

[Horusprg]

check out the demo, you might want to use dynamic imports. Now the webgazer is loading as a second content script. https://github.com/koll93/extension-manifest3-webgazer

I used the webgazer.js from your example in a extension from a project what i'm developing and worked fine to me, i puted the full code inside the main.js code and works https://github.com/Colab-UNIRIO-UFPA/UX-Tracking-Extension

[jeffhuang]

been testing this. hopefully will merge today. thanks for your patience