Create a mock WebExtension API for developers to run unit tests with

[kumar303]

Use case: "As a WebExtension developer, I'd like to unit test my extension without launching a real web browser. I'd like to use a mock library to make assertions about my use of the WebExtension API"

(This issue would likely have nothing to do with the web-ext tool itself, I'm just not sure where else to track it.)

This library would let you interact with a mock WebExtension API and make assertions about the results. It would probably use something like sinon for the mocking part.

Schemas

The library should use schema files to fully sync up with all APIs as well as future APIs and API changes. This will make the mock API a very precise representation of the real API. Here are Firefox's schema locations:

• Internal toolkit APIs: http://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas
• Browser UI APIs: http://searchfox.org/mozilla-central/source/browser/components/extensions/schemas

These schemas are identical to Chrome's but in Firefox the namespacing is a bit different (browser, not chrome) and the asynchronous functions return promises, not callbacks. Note that some Chrome schemas are IDL files but most are JSON schema files.

Prior art

There are a few libraries but they are Chrome only and are either incomplete or don't fully sync with schemas:

• chrome-mock
  • uses schemas but seems incomplete
  • schema files are copy/pasted, not in sync with upstream source code
• sinon-chrome
  • uses some schema files, may not be in sync with upstream source code

If it's easier to patch an existing library, let's do it!

[Standard8]

Note the use of schemas was suggested to sinon-chrome in https://github.com/acvetkov/sinon-chrome/issues/40

[acvetkov]

@kumar303 @Standard8

sinon-chrome use chrome schema files since v.2.0. https://github.com/acvetkov/sinon-chrome/tree/master/src/config

Implementing FF schema files in progress.

[kumar303]

@acvetkov I did see that (I will edit my description, oops). I was wondering, how often are those synchronized with the Chromium source code?

[acvetkov]

I will update as needed, for example, on new new API release.

[kmaglione]

Collaborating with sinon-chrome sounds like a good option. And possibly combining it with mozilla/webextension-polyfill

[arantius]

I'm trying to do unit testing on a webextension. An example which demonstrates how to connect a testing framework, sinon-chrome (maybe?) and my source (which calls many webextension APIs) together would be quite helpful.

[Standard8]

I'm trying to do unit testing on a webextension. An example which demonstrates how to connect a testing framework, sinon-chrome (maybe?) and my source (which calls many webextension APIs) together would be quite helpful.

I have one such example here:

https://github.com/Standard8/example-webextension/

[arantius]

Thanks @Standard8 . I'm trying to stitch too many things together at once I guess, because npm and the javascript ecosystem these days. Sinon and Karma and Mocha and Chai, oh my.

So if I check out that example and npm run test:karma it passes. If I insert a chrome.runtime.sendMessage().then() it fails, because (I think) I'm getting the Chrome APIs (which accept callbacks), not the Firefox APIs (which return Promises).

Maybe I should be referring straight to browser? But then I get browser is not defined. I can find a claim ( https://github.com/acvetkov/sinon-chrome/issues/40 ) that sinon-chrome supports WebExt, but no information on how to use such support.

[serv-inc]

Hi @arantius, it seems like you can (always?) use the Firefox APIs like the chrome ones, passing in a callback.

[arantius]

I don't want to. I want to use promises, they make for more readable code. But I still want to have tests of that code.

[stoically]

I've started working on a WebExtensions API Fake (mimicking actual implementation) based on sinon-chrome here: https://github.com/stoically/webextensions-api-fake

It's really not much at this point and mainly focused on the needs of feature-testing multi-account-containers, so, if you want some specific API supported, feel free to get in touch.

@arantius maybe the included example (which is also available as runnable version in the test-directory) helps in your case, if you haven't solved it meanwhile.

[kumar303]

@stoically thanks for posting a comment about it. Out of curiosity, what are the limitations of just using sinon-chrome directly? It looks like it already supports the contextualIdentities API.

[stoically]

@kumar303 It's a (rudimental) Fake implementation of the API (currently supporting contextualIdentities, tabs and storage.local) that makes it possible to feature-test WebExtensions browser-free without having to manually instruct the stubs to do certain behavior. This is mostly interesting for feature-tests where manually faking all the API behavior on stubs can become cumbersome. For unit-testing one would probably still want to manually fake stub behavior (by just using sinon-chrome). So, it's actually built on top of sinon-chrome and can complement it, not meant to replace it.

I'm currently using it to fake-stub the browser in background and popup htmls using only nodejs with jsdom, which lets me simulate a click in the popup and check if certain calls on the background browser are made (e.g. contextualIdentities.create). Here's an example.

I'm also interested in the possibility to use web-ext in combination with test-frameworks to execute (mocha) tests in the actual browser-environment from nodejs, but I guess that's a tougher task. I think @Standard8 worked on something that could/can accomplish that (without web-ext) at https://github.com/Standard8/example-webextension, though it's archived now.

[balta2ar]

Hello. I'm developing a WebExtension that uses Native Messaging. I'd like to be able to write some kind of functional/integrational tests. My Native Messaging App has a REST API and it acts as a mediator between my Client App and the WebExtension, so it's probably the only entry point I need to test (I interact with the extension code through the App only). What's the best way to write such tests in the current state of things?

[stoically]

@kumar303 I've created webextensions-jsdom and included an example webextension + test to illustrate how the several parts play together. Maybe that helps showing how that can be useful.

@balta2ar If you want to make an actual integration-test that you can start from nodejs with functional native messaging available, then that's something I'm looking for as well. What I found so far regarding executing test in the actual WebExtension context in the browser is "Running tests from within the addon" in this official example. However, that only lets you execute tests encapsulated in the browser, you can't wire it up with your client app or execute it from nodejs (and get actual browser context exposed) - at least I'm not aware of how that could be possible - although it looks like the already mentioned example-webextension has some code already doing exactly that.

What I imagine is something along the lines of

const webExt = require('web-ext/api');
webExt.loadTemporaryAddon('manifest.json');

Now webExt exposes e.g. .popup and .background, both also with the window and window.browser property. That would make it possible to easily execute feature/integration tests from nodejs within the actual browser environment. Though, maybe that's already possible and I just missed it, because loading web-ext from NodeJS is already possible, it just don't mentions if/how window/browser contexts from popup/background are exposed.

[Standard8]

@stoically The only reason example-webextension is archived is that I decided not to keep on updating it just for package updates. If it doesn't work out of the box today, then there's probably only a few minor tweaks needed.

As you've already pointed out, the unit tests use sinon-chrome to stub out the various WebExtension APIs - because it knows about the schemas, it can stub them all out.

The functional tests are based on running with Mocha (via node), and running Firefox with the WebExtension via Selenium. This has the advantage that you can get the test code to interact with external items as well as driving Firefox with the WebExtension built in. I've done this kind of interaction on previous projects, and it worked quite well.

[stoically]

@Standard8 just tested and it works just fine out-of-the-box. Having mocha/node handle running Firefox via Selenium and loading the Temporary Add-on that way is indeed really handy. It makes for great functional/integration tests - thanks for putting it out there! Do you by chance know a way to access the popup/background/contentscript-specific window (DOM) in this case?

@balta2ar So in case your Add-on has observable behavior in the Firefox UI (or by selenium / webdriver), the mentioned example-webextension should fit your needs.

[stoically]

@balta2ar Published webextensions-geckodriver on npm. It's just a small wrapper around the work @Standard8 (thanks again!) put into example-webextension - maybe it helps you with testing your WebExtension.

[motin]

Do you by chance know a way to access the popup/background/contentscript-specific window (DOM) in this case?

I only know of one way for the tests to access background-privileged APIs, and it is not optimal: Let the extension's background script open up an extension page, switch to that tab/window and execute javascript using driver.executeAsyncScript().

Check this helper method for more information and an initial implementation: https://github.com/mozilla/shield-studies-addon-utils/blob/develop/testUtils/executeJs.js

Tests can then be written as such:

  it("should be able to access window.browser from the extension page for tests", async () => {
    const hasAccessToWebExtensionApi = await utils.executeJs.executeAsyncScriptInExtensionPageForTests(
      driver,
      async callback => {
        callback(typeof browser === "object");
      },
    );
    assert(hasAccessToWebExtensionApi);
  });

This way, WebExtension APIs available only to background scripts can be used directly in tests.

[gkrishnaks]

Will be great if someone can clone https://github.com/Standard8/example-webextension and provide a sample for running test cases in headless firefox in some CI CD runner.

I tried that once from gitlab CI CD, I will look into it more and update here. cc @satwik163

[Lusito]

If anyone is interested, I started a new approach, feel free to take a look and give feedback.

Here's the discussion on discourse: https://discourse.mozilla.org/t/introducing-mockzilla-and-mockzilla-webextension-for-easy-web-extension-testing-with-jest/59159

These are the projects: https://lusito.github.io/mockzilla-webextension/ https://lusito.github.io/mockzilla/