motin
commented
3 years ago Revisited some of this to check where we are at today:
Mocking the web extension API:s
Up to date definitions and an ability to request mocks for different versions of Firefox would be helpful.
Have not revisited this, since I have not had much need to mock the web extension APIs.
On a related note, up to date TypeScript definitions are available when using https://github.com/Lusito/webextension-polyfill-ts, which is anyhow recommended when writing cross-browser extensions.
Testing using a real browser instance
Less boilerplate
A set of re-usable Selenium test methods (like https://github.com/mozilla/shield-studies-addon-utils/tree/master/testUtils) for high-level browser automation (setting up profiles, starting/restarting the browser, installing web extensions etc) makes writing these tests easier.
Puppeteer now officially supports Firefox (albeit it is still considered experimental), and can be launched with specific extensions installed. However, it is not possible to interact with the browser UI, meaning that we can't instruct Puppeteer to click the extension's browser icon for instance. (source)
Since this is pretty important when functionally testing Web Extensions, the re-usable Selenium test methods strategy still seems the most relevant.
Access to the extension background context
I found a way last year to access the Web Extension background context using Selenium, but while it works, it is rather clunky and slow. While it is still the current best practice for add-ons outside the tree, puppeteer-firefox may soon become a viable alternative, allowing direct access to extension background context (as well as speedier functional tests and a better development experience in general).
Unfortunately, while the Chrome DevTools Protocol supports switching to the extension context, Puppeteer itself does not expose this functionality yet. There is a two year old fork of Puppeteer that exposes this (mentioned here) but such an old version of Puppeteer is lacks Firefox support and other important recent improvements.
Thus, even with Puppeteer, one is constrained to add an extension page to the extension and navigate to it in order to expose an extension background context.
Code coverage
I have found no examples of code coverage instrumented that works with Web Extension tests when a real browser is used, but at least https://theintern.io seems to be able to instrument code coverage even when running tests via Selenium so maybe there is some hope...
Have not investigated this further, only verified with theintern developers that Web Extensions are not supported. It may still be relevant to check how/if they instrument codecoverage for Selenium-run tests.
Performance testing
Measuring impact of add-ons on web performance by using exthouse is discussed in https://discourse.mozilla.org/t/analyze-add-ons-performance-impact-on-a-user/41813 (presentation of the exthouse add-on from June 27 2019: https://mzl.la/add-ons-demos-2019-06-27) and may become available for Firefox extensions.
This is still not available for Firefox: https://github.com/treosh/exthouse/issues/26
I run into this all the time when writing web extensions for Firefox:
I'd love to be able to take smaller units of code in my add-on and write tests against it, but if those smaller units of code use any web extension API, ordinary JavaScript/Typescript testing tools fall short since the APIs are not available in the standard JS context.
This repository should contain a modern example of automated testing of web extensions that overcomes this constraint.
State-of-the-art best practices on overcoming the lack of web extension APIs in the standard JS context today
Mocking the web extension API:s
Using something like sinon-chrome (see this MDN example) may be viable and seems to be best practice, but I have found that the definitions used to provide the mocks are often out of date or only available for a single major version of Firefox at a time.
Testing using a real browser instance
Many times the code that I want to test rely on the functionality of actual web extension APIs. From what I have gathered so far, best practice here is currently to use Mocha or similar together with Selenium for writing functional browser-level tests (as in https://github.com/mozilla/shield-studies-addon-template/tree/master/test/functional), but it involves quite a lot of boilerplate code, does not easily allow access to the extension background page, does not allow for code coverage instrumentation and does not include any performance-related testing.
Possible improvements on today's state-of-the-art
Mocking the web extension API:s
Up to date definitions and an ability to request mocks for different versions of Firefox would be helpful.
Testing using a real browser instance
Less boilerplate
A set of re-usable Selenium test methods (like https://github.com/mozilla/shield-studies-addon-utils/tree/master/testUtils) for high-level browser automation (setting up profiles, starting/restarting the browser, installing web extensions etc) makes writing these tests easier.
Access to the extension background context
I found a way last year to access the Web Extension background context using Selenium, but while it [works](), it is rather clunky and slow. While it is still the current best practice for add-ons outside the tree, puppeteer-firefox may soon become a viable alternative, allowing direct access to extension background context (as well as speedier functional tests and a better development experience in general).
Code coverage
I have found no examples of code coverage instrumented that works with Web Extension tests when a real browser is used, but at least https://theintern.io seems to be able to instrument code coverage even when running tests via Selenium so maybe there is some hope...
Performance testing
Measuring impact of add-ons on web performance by using exthouse is discussed in https://discourse.mozilla.org/t/analyze-add-ons-performance-impact-on-a-user/41813 (presentation of the exthouse add-on from June 27 2019: https://mzl.la/add-ons-demos-2019-06-27) and may become available for Firefox extensions.