Test System for v6

[gregglind]

• [ ] decide test system
• [ ] describe how in readme
• [ ] implement (may get new bug)

[gregglind]

Issues:

• web-ext only sort of works for re-using profiles.
• web-ext sets too many settings
• selenium / webdriver is different than treeherder / usual mozilla ci.

[motin]

I looked into using the same testing setup that the official web extensions use, however it would require us to copy over source files from here into the firefox source tree, and doing so would not let us catch the specific issues associated with bundling web extension experiments. Since we will use selenium for functional testing (or is there any alternative that we should rather use?) it makes sense to also use selenium for testing our experiment APIs.

The test add-on currently accomplishes this by opening an extension page, switching to that tab/window and executing javascript using driver.executeAsyncScript().

Check this helper method for more information and an initial implementation: https://github.com/mozilla/shield-studies-addon-utils/blob/develop/testUtils/executeJs.js

Tests can then be written as such:

  it("should be able to access window.browser from the extension page for tests", async () => {
    const hasAccessToWebExtensionApi = await utils.executeJs.executeAsyncScriptInExtensionPageForTests(
      driver,
      async callback => {
        callback(typeof browser === "object");
      },
    );
    assert(hasAccessToWebExtensionApi);
  });

This way, WebExtension APIs available only to background scripts (including experiments) can be accessed directly in tests.

[gregglind]

Currently done, but we want to move to mochi test.

[motin]

Cross-posting this excellent post about tests from @rhelmer in https://github.com/mozilla/shield-studies-addon-utils/issues/173#issuecomment-388213835:

webExtension Experiment API yes to linting, prettify yes to selenium / webdriver

sgtm, for webextension code (see below re: experiments code which I think is different)

no to mochitest, and I have no idea of how to do that no to talos perf and I have no idea of how to do that.

@biancadanforth and I were just discussing this today... we are working with Mozilla infra folks to make this pretty seamless.

We're also going to look into the feasibility of running more standard test frameworks like Mocha, I don't think there's much point in writing mochitests that specifically test study code.

It's not all fleshed out yet, but the back-of-napkin sketch so far with regard to test integration is something like this:

1) opening a PR causes a travis/circle/something run, which runs any study-specific code using mocha and linting and such, and also downloads firefox and runs a basic subset of the test suite 2) less frequently (maybe on merge to master?) a taskcluster job that runs on real mozilla infra will be kicked off. This will run correctness (function/integration, unit, and perf tests) with the shield study installed to assess how the study impacts any existing Firefox features and performance.

We hope that this will make it unnecessary for shield study authors to have to interact much with the Mozilla-specific test harnesses and infrastructure - the exception is going to be if we need to dig in to understand why some Firefox test is failing, but this should be the exception rather than the rule.

I also expect that as we build a well-tested set of experimental webextension APIs, we'll compartmentalize this sort of work there and individual study authors will be writing pure webextension code that may use these experimental APIs, and won't necessarily be solely responsible for tracking down anomalies.

Note though that for experimental webextension APIs, I think we should use the Mozilla test harnesses like xpcshell and mochitest and so on, since this is essentially Firefox code and it needs special tool support that we already have. Also, it's likely that very useful experiments could be landed in Firefox proper and that will help this.

[motin]

@gregglind Vote for renaming this issue to target v6 instead of already release v5