chore(web): karma is deprecated, find an alternative test runner

mcdurdin commented 5 months ago

See also #10494

jahorton commented 5 months ago

https://github.com/karma-runner/karma#karma-is-deprecated-and-is-not-accepting-new-features-or-general-bug-fixes

jahorton commented 2 months ago

Sadly, as of this moment at least, BrowserStack's tutorials for establishing JS unit testing still utilize Karma.

https://www.browserstack.com/docs/automate/javascript-testing/getting-started

That is, there's no quick and easy reference for migrating to an alternative approach compatible with their service.

jahorton commented 2 months ago

One of the leads from Karma's deprecation page - Web Test Runner - does have a BrowserStack plugin: https://modern-web.dev/docs/test-runner/browser-launchers/browserstack/

Web Test Runner says:

For modern browsers, we recommend using other browser launchers, as they are a lot faster. Browserstack is a good option for testing on older browser versions.

Looking into some of its other recommendations, a testing engine called Playwright is notably compatible. It's very focused on running the "latest and greatest", though - it won't be helpful for testing legacy browsers. That said, it could allow us to offload our "modern browser" section of tests onto the build agents, bypassing any Browserstack connectivity issues. It also has full Chromium, WebKit, and Firefox-engine support - runnable from all major platforms. Apparently, it can run tests under simulated mobile environments with WebKit and Chromium, too.

The biggest issue I can see: it looks like it'll take some work to convert from the Karma test-specification format. But... that's just the nature of tech-debt, I guess.

I suppose the option always did exist with Karma, but I just realized that it's totally possible to make one config that sets up Playwright launchers and BrowserStack launchers - it's not like we'd have to specify only one. It's not clear how it handles cases where BrowserStack launching fails to connect yet, though.

jahorton commented 2 months ago

Extra thing of coolness for Web Test Runner: https://modern-web.dev/guides/test-runner/code-coverage/#coverage-browser-support

Runs against Chromium can give code coverage reports, and it does seem like the format is either c8-compatible directly or easily convertable to the same (based upon istanbul): https://modern-web.dev/docs/test-runner/writing-tests/code-coverage/

jahorton commented 2 months ago

A thought occurs: we have regular user testing now. Not only do we already have user & regression tests against Android 5.0 via Android Studio emulation... but BrowserStack does offer live testing, not just automation. We could just schedule semi-regular "test this live via BrowserStack" user tests / regression tests against specific legacy targets in order to validate that things still work on older devices. Essentially, it'd result in less-frequent legacy testing but still give us a semi-regular 'canary' for legacy-compat issues. From there, we could shift to only automatically testing against modern targets.

I'm not saying it's ideal, but... it would certainly be a practical decision, if naught else, and keeping the tests fully on our build agents should certainly help with flaky connectivity issues.

jahorton commented 2 months ago

Another caveat: there doesn't seem to be a public teamcity-reporter for Web Test Runner. That said, they've got decent docs on how to DIY one, so it's not the end of the world.

If we're willing to take a stronger shift in testing paradigm, use of playwright directly can come with some notable benefits. One really interesting one: automatic test generation via recording.

https://playwright.dev/docs/codegen#introduction
We could then write the tests with Mocha as the runner, issuing commands via playwright to execute tests.
Mocha has a pre-existing teamcity-reporter, so test-output formatting wouldn't be a big concern.
Playwright can also connect for code-coverage stuff: https://playwright.dev/docs/api/class-coverage
They have some fairly strong VS Code integration via plugins; it's even possible to use VS Code as the code inspector, it seems.

Huh. And, it turns out that playwright... is published on GitHub by Microsoft.

They also have their own mocha-like test-definition structure we could use instead of driving things within mocha test-specs, though that of course means we hit the TC-reporter issue again. That said, their structure does make some nice guarantees about making sure the document is fully settled at each step, removing a lot of flakiness issues we've seen at times.

mcdurdin commented 2 months ago

Side Note: Karma sometimes eats all available memory on our build agents, for example today on ba-win10-64-s1-601:

Name    PID Status  User name   CPU Working set (memory)    Memory (active private working set) Commit size Command line    UAC virtualization
node.exe    8028    Running bob 03  4,540 K 4,372 K 1,156,644 K "C:\Program Files\nodejs\node.exe" C:\BuildAgent\work\99b311828f4ee7c\keyman\node_modules\.bin/../karma/bin/karma start --log-level=debug --reporters teamcity,BrowserStack ./in_browser/CI.conf.cjs    Not allowed

keymanapp / keyman

chore(web): karma is deprecated, find an alternative test runner #10497