Integrate mocha (and electron-mocha) testing framework

[rdw-software]

There's so many, I don't even know which to pick. An overview: https://www.lambdatest.com/blog/top-javascript-automation-testing-framework/

Goals:

• [x] Tests should be able to run optionally as a separate "mode" (i.e., injected into the load process), without cluttering the environment if run normally
• [x] Addons should be able to register their own tests, to be run alongside the engine ones (in both modes)
• [x] Tests should be executed on each commit (via GitHub actions)
• [x] Feedback should be given on failing tests for Pull Requests (via GitHub actions bot?)
• [ ] It should be possible to run both engine and addon tests from inside the app and display a report window

Roadmap:

• [x] Add mocha and electron-mocha npm dependencies
• [x] Add npm script to run the tests
• [x] Add a shortcut (cmd) to run the tests via CLI
• [ ] Add a shortcut (cmd) to run the tests in the browser (HTML report)
• [x] Add a C_Testing API to collect functions related to running and setting up the tests
• [ ] Add a way to load all the mocha stuff on demand, via C_Testing.enterTestMode() or similar
• [ ] Add C_Testing.showReport() to show the latest rest report
• [ ] Introduce a TEST_RUNNER_STARTED event to allow addons to hook into the run
• [ ] Trigger this event when the electron-mocha runner starts or the client is started in test mode
• [x] Update GitHub actions config to run tests on commit

[rdw-software]

After looking into this, here's my preliminary conclusion:

• https://jestjs.io/ - Recursively walks the file system with no way to turn it off, even for single files (absurdly slow). Not very delightful
• https://mochajs.org/#running-mocha-in-the-browser - Liked it. The spec-based syntax is awkward but it could be wrapped. In-browser tests are not repeatable, and only god knows why
• Other options: Nothing stood out

The unfortunate part is that the APIs are exposed as globals (by design, similar to the WOW client's Lua environment it was modeled after). That doesn't work with offline testing, so in-browser tests are the best approach.

This isn't an issue since the client doesn't do anything that would affect the tests on its own, assuming no addons are installed and no APIs called/UI events fired. I found https://github.com/jprichardson/electron-mocha to make up for this and was able to run tests in the render thread (no UI), which could be automated in a pipeline as follows:

electron-mocha --url index.html --renderer <path to test file or directory>

This is looking like the best bet so far, and could possibly be combined with "regular" mocha for displaying test reports in-app (manually started), if the re-run issue is resolved. That is, in the CI pipeline or CLI we use electron-mocha, and for unit tests or in-browser visualization simply mocha. Electron is needed for integration and system tests anyway, and it's better to use the same framework/syntax for both instead of having to write a custom testing framework.

[rdw-software]

Awkwardly, I managed to make the tests run repeatedly. It requires a ton of hacks and I'm not a fan of this approach. Even then, a few problems remain while running mocha in the browser directly:

• For some reason, it appears to hijack the global environment and causes WebGL "context lost" errors with BabylonJS
• The UI needs to be fixed manually as they continually append divs to it after running tests, and the progress indicator is just gone after the first run (causes inconsistent UX)
• When clicking on the individual test suites, mocha will reload the entire app (!!!) to run them again, and this apparently can't easily be changed

All things considered, I see only two viable options:

• Stick with mocha and mocha-electron, but create a separate "test runner" version of the app (or a way to load tests with some flag only, that could be triggered via CI or CLI)
• Create a custom runner that works just as I want it to (too much irrelevant work/reinventing the wheel...)

Too bad they didn't account for this use case, but it might be easiest to just run tests visually in a separate config if needed.

[rdw-software]

Honestly, this framework just gives me one headache after the other when it comes to running tests in the browser. I think I'll just remove that feature and settle for CLI, as that at least works reliably. If we ever need more, it's probably time to write our own framework unless there exists one that doesn't behave erratically... :weary: