Cross browser testing

[holtchesley]

Create cross browser/os testing using a service like BrowserStack or SauceLabs. As of 2022-02-01, it looks like BrowserStack and SauceLabs make this easy to do if we decide to use javascript for the (playwright) testing. So right now we've got a few options:

1. Use JavaScript and Playwright to write some tests.
2. Use Python and Selenium to write some tests.
3. Dive into getting python-playwright working for one of these services.
4. Wait a bit for the service(BS/SL) to do number 3 for us.

[holtchesley]

We'll shortly have a bit of cross browser testing (Chromium, Firefox, Webkit) on Linux. Option 4 seems like the best option for at least the short (~months) future to me.

[matteocargnelutti]

Update: our Playwright tests are Chromium-only at the moment, as a result of the extremely complex setup required to get local SSL certs working in this context. It's not clear yet whether or not @rebeccacremona will be able to pull that feat off once again for Firefox, nor if it's necessarily worth the effort.

There has been talks of separating our E2E testing into two "suites":

• 1 chromium-only suite using Playwright, focused on mission critical items, which can be run locally and on PR.
• 1 cross-browser suite using Browserstack (or similar technology), running tests against a given deploy for less crucial items.

TBD.