How to use WebGazer in a browser extension?

[rifazn]

I am trying to use WebGazer in a browser extension. The extension loads up webgazer.min.js as the very first script and then checks to see if WebGazer is ready in the next script. This is the contents of the manifest.json file:

manifest.json:

{

  "manifest_version": 2,
  "name": "Gaze to Surf",
  "version": "1.0",

  "description": "Turn your webcam on and look at weblinks to browse the web!",

  "icons": {
    "48": "icons/webcam_iconfinder.svg",
    "96": "icons/webcam_iconfinder.svg",
    "128": "icons/webcam_iconfinder.svg"
  },

  "web_accessible_resources": [
      "content_scripts/webgazer/*"
  ],

  "content_scripts": [
    {
      "matches": ["http://localhost/*", "<all_urls>"],
      "js": [
          "content_scripts/webgazer/webgazer.min.js",
          "content_scripts/startgaze.js"
      ]
    }
  ]

}

And startgaze.js:

window.applyKalmanFilter=true
window.saveDataAcrossSessions=true

function checkIfReady() {
    if (webgazer.isReady()) {
        webgazer.setRegression("ridge")
            .begin()
            .showPredictionPoints(true)
    }
    else {
        console.log("WebGazer not ready")
        setTimeout(checkIfReady, 100)
    }
}
setTimeout(checkIfReady, 100)

// window.onbeforeunload = webgazer.end

When I load this extension in either Mozilla Firefox, or Google Chrome, checkIfReady method keeps executing as webgazer.isReady() just keeps evaluating to false.

So the questions I have are:

1. How do I start WebGazer so that it works properly in a browser extension?
2. The wiki states that WebGazer should be loaded from a server. But to develop a Web Extension, all the scripts are required to be present locally. How should I work around this?
3. I have the Source Map files in the same directory as the webgazer.js and webgazer.min.js files but Chrome reports in console that it cannot find the Source Maps still. This is the exact console output:

DevTools failed to load SourceMap: Could not load content for chrome-extension://<some id internal to chrome>/content_scripts/webgazer/webgazer.min.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME

I am using version 2.0.1 of WebGazer (latest at time of posting), and have built the project from the instructions.

In case its required to see the whole directory structure, I have also hosted the whole codebase here: https://github.com/rifazn/WebGazerExtensionTest

[xanderkoo]

/**
 * Checks if webgazer has finished initializing after calling begin()
 * [20180729 JT] This seems like a bad idea for how this function should be implemented.
 * @returns {boolean} if webgazer is ready
 */
webgazer.isReady = function() {
    if (videoElementCanvas === null) {
        return false;
    }
    return videoElementCanvas.width > 0;
};

This is the code for the webgazer.isReady() function, and I agree with the comment that this may not be the most suitable way to implement it, especially for your use. You might try and redesigning it to fit your needs.

I think your issue with net::ERR_UNKNOWN_URL_SCHEME seems to be unrelated to WebGazer specifically. @Skylion007 any thoughts?

[rifazn]

Ah. But then the question that arises is, why isn't the videoElementCanvas getting initialized?

[xanderkoo]

I think it might be worthwhile to look at the structure of www/js/main.js. It configs and initializes webgazer before checking if it's ready. I think you would have to call webgazer.begin() before videoElementCanvas gets initialized, but experiment with that a bit and see if it works.

[rifazn]

Ah, yes. You're right. That is the case. I had originally done it like that but down the line something ended up not working and my late night brain wanted to try that as a confused and last ditch effort.

I'll report back the effects after removing checkIfReady. This is my updated content_scripts/startgaze.js however.

[rifazn]

Changes after last comment:

I have removed that checkIfReady now and also changed the extension so that it only runs when a button on the browser toolbar is clicked. To do that, I added a browser_action (the toolbar button) in the manifest and a background_script and a content_script.

The background.js adds a click listener to the extension's toolbar button. When that button is clicked, it loads up webgazer.js and executes the code inside startgaze.js. Upon successful consecutive execution of the scripts it sends messages (saying scripts loaded successfully) to content.js which just prints to console the messages sent by background.js. startpage.js executes webgazer.begin() with preferred options.

Note: browser-polyfill.js is added to make the extension work in Google Chrome as it does not support the Promise-based Browser Extension APIs out of the box.

The Problem that exists:

It seems to work as expected in Chrome. The WebGazer library loads up and after calling webgazer.begin(), the video overlay is shown, facial features detected and predictions are also made. But on Firefox, the video overlay is only shown but nothing works after that. The facial features are not detected and WebGazer makes no predictions at all.

I have pasted the code here for convenience, but you can also find the current code in this GitHub link.

This is the code as it stands right now:

manifest.json:

{
  "manifest_version": 2,
  "name": "Gaze to Surf",
  "version": "1.0",

  "description": "Test WebGazer in a browser extension.",

  "icons": {
    "32": "icons/webcam32.png",
    "48": "icons/webcam48.png",
    "96": "icons/webcam96.png",
    "128": "icons/webcam128.png"
  },

  "permissions": [
    "tabs", "activeTab"
  ],

  "browser_action": {
    "default_icon": "icons/webcam32.png",
    "default_title": "Gaze to Surf!"
  },

  "background": {
    "scripts": [
        "content_scripts/browser-polyfill.js",
        "background.js"]
  },

  "content_scripts": [
      {
          "matches": ["<all_urls>"],
          "js": [
              "content_scripts/browser-polyfill.js",
              "content_scripts/content.js"]
      }
  ]

}

content_scripts/content.js:

browser.runtime.onMessage.addListener(message => {
    if (message.message)
        console.log(message)
})

background.js:

function sendMessage(tabID, messageObj) {
    return browser.tabs.sendMessage(tabID, messageObj)
}

/*
On browser action click, load WebGazer in the current tab and run it
*/
browser.browserAction.onClicked.addListener(() => {

    var gettingActiveTab = browser.tabs.query(
        {active: true, currentWindow: true}
    );

    gettingActiveTab.then((tabs) => {
        sendMessage(tabs[0].id, {
            message: 'Extension started.'
        })
    })

    browser.tabs.executeScript({
        file: "/content_scripts/webgazer/webgazer.min.js"
    })
    .then(() => {
        gettingActiveTab.then(tabs => {
            sendMessage(tabs[0].id, {
                message: "WebGazer library loaded."
            })
        })
    })
    .then(() => browser.tabs.executeScript({
        file: "/content_scripts/startgaze.js"
    })
    )
    .then(() => {
        gettingActiveTab.then(tabs => {
            sendMessage(tabs[0].id, {
                message: "stargaze.js has run."
            })
        })
    })
    .catch((reason) => {
        console.error(reason)
        gettingActiveTab.then(tabs => {
            sendMessage(tabs[0].id, {
                message: "Could not execute Webgazer. Reason: " + reason
            })
        })
    })
})

startgaze.js:

window.applyKalmanFilter=true
// window.saveDataAcrossSessions=true

webgazer.setRegression("ridge")
    .begin()
    .showPredictionPoints(true)

document.addEventListener("keydown", ev => {
    if (ev.ctrlKey && ev.key == ">") {
        webgazer.end()
    }
    else if (ev.ctrlKey && ev.key == "?") {
        webgazer.end()
        webgazer.clearData()
    }
})

// window.onbeforeunload = webgazer.end

[fxnoob]

Hi @rifazn! Using webgazer script as a Content script is not the good way to use it in a browser extension. As you have to grant the permission on every page to script to use the camera(that way actual website from which user allows the script to use camera, can use the camera input). Best way could be a Option Page in which you include the webgazer script in a regular script tag and use it, so you only have to allow camera access one time(there is no need to use localhost or https for this). you can use predicted coordinates in current page with the help of Message Passing between current active tab and option page.

FYI: Option page behaves as a regular background script(so you Get to access all the browser extension apis on this page as well). If you are interested in a react based setup for browser extension (chrome). you can use this boilerplate https://github.com/fxnoob/chrome-extension-boilerplate

and add this dependency

npm i 'webgazer'

and then build a option page with it.