search

shaka-project / shaka-player

JavaScript player library / DASH & HLS client / MSE-EME player
Apache License 2.0
6.93k stars 1.31k forks source link

Karma is deprecated #5547

Open joeyparrish opened 10 months ago

joeyparrish commented 10 months ago

I just found out that Karma was officially deprecated 4 months ago. https://dev.to/this-is-angular/angular-testing-in-2023-past-present-and-future-j5m

This isn't urgent, but we should decide what our test runner will look like in the future.

Things we need:

  1. Real browser testing through Selenium grid so we can test video playback on real devices (not using jsdom or phantomjs)
  2. Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser (this gets us the exotic devices in our lab through generic-webdriver-server)
  3. Serve tests over HTTPS with a custom certificate (without which we lose EME coverage and encrypted playback tests in the lab)
  4. A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)
  5. Machine-readable test coverage reports

Things that would be make the transition easier:

  1. Support for our current test framework, Jasmine (though we could rewrite our tests if necessary)
  2. Test coverage output in the same JSON format we parse today in our workflows (though we could hypothetically convert to that or rewrite our workflows)
  3. A way to run local browser tests over WebDriver, maybe with a plugin (this is how we get screenshot tests without using the lab today, via karma-local-wd-launcher)

Input from the community is very welcome. I've done very little research so far.

joeyparrish commented 9 months ago

I believe jasmine-browser-runner is the way forward. It will let us keep our existing Jasmine tests, it will connect to a remote Selenium grid, and we can reimplement our custom middlewares in express via configuration.

I'm not yet sure about:

  1. ~TLS configuration for testing over HTTPS~ Resolved by my PR, https://github.com/jasmine/jasmine-browser-runner/pull/49
  2. Launching local WebDriver instances to support local screenshot testing (probably could do it by starting with something like this)
  3. Coverage reporting
joeyparrish commented 3 months ago

jasmine-browser-runner also seems to only run a single browser. That will also need to be addressed.

joeyparrish commented 1 month ago

The maintainers of jasmine-browser-runner are resistant to adding support for multiple browsers running in parallel, even if I provide a PR. (And I had this working locally.) They just don't seem to want the complexity that they feel would come with that functionality. So we may have to go another direction.

I'm currently investigating Web Test Runner, which is built to launch real browsers, just like Karma was. It supports both Selenium and WebDriverIO as WebDriver clients to launch browsers.

Needs met by Web Test Runner:

Needs, not sure how to get from Web Test Runner:

Wants: