search

dgp1130 / blog

Source repository for my personal blog.
https://blog.dwac.dev
3 stars 2 forks source link

Remove Karma #80

Open dgp1130 opened 4 months ago

dgp1130 commented 4 months ago

With Karma's recent deprecation, I should move the blog to something else. I do feel strongly that I want to use a real browser for browser tests, which excludes options like Jest. I think the two potential replacements are Web Test Runner and Vitest's new experimental browser support.

I've had good experiences with Web Test Runner in the past and it can support Jasmine (which I'm already using) though it doesn't support that out of the box and does take some effort to connect everything. I haven't played with Vitest in this context though so it might be worth trying out there.

dgp1130 commented 4 months ago

Played around with Vitest browser mode and have my attempt in ref/vitest. It does mostly work just fine, I didn't run into any particular infra bugs while using it. However I think I'm not going to move forward with it for the moment. I'll leave some notes here for future reference:

Browser UI

The biggest problem is that tests are not debuggable. While I can open tests in regular Chrome, I can't breakpoint in source code or inspect the environment. Disabling JS sourcemaps did not help. Sometimes I can find the TS/JS files, sometimes I can't. When I do find them and put a breakpoint, the breakpoint is never hit and outright dropped on refresh. This is probably my biggest issue right now as I have a long standing rule of never adopting a test runner unless I am able to effectively debug tests, not just run them.

The browser UI is not particularly smooth right now. The "Browser UI" tab is empty for me. Clicking on a test has no effect. I have to open test source and then switch to results to see the test error. I assume this is just the experimental status and the UI will be improved over time, I really don't care that much, it just feels kinda broken atm.

Interestingly I'm not seeing any iframes executing the actual test code, yet the document appears to be empty when inspecting document.body in the test. So I'm curious to understand how the browser UI is shown without affecting the tests. I'll have to look a little deeper once I can actually inspect a script.

Configuration

I was ultimately able to make what I wanted work, but had some struggles along the way and needed to tweak the configuration a bit.

vitest starts a devserver in watch mode with the web UI and opens the vendored browser controlled by automation. Personally I hate that UX as I prefer to keep test results in the terminal where possible and the automated browser doesn't match the configuration of my personal browser (DevTools doesn't include Console open by default, tabs are in the wrong order, no retained settings, etc.) Instead, I set up the NPM script to use vitest run to do a one-off execution with headless: false by default to stick to the terminal. I have a separate debug command which does vitest watch --browser.headless false --browser.provider preview. This opens the web UI with my system browser (including all my preferences) which I think is much more ergonomic to debug. Technically there might be some version differences between the vendored browser and my installed browser, but IMHO that's so rarely a problem that I really don't care and the UX ergonomics are more important.

TypeScript setup was quite annoying. Vitest seems to parse and transpile TypeScript automatically, even with no configuration. However it does not type check by default, something I was already aware of. I immediately reached for the typecheck options before discovering they have to do with testing types, not running typechecking on your test code. SEO is a real pain with solving the second problem as I could not get vitest to fail for a type check error.

I eventually discovered vite-plugin-checker and set that up to run type checking as part of the test, however this has some caveats.

Firstly, I'm not a fan of the terminal output. Type errors look like this:

 ERROR(TypeScript)  Element implicitly has an 'any' type because type 'typeof globalThis' has no index signature.
 FILE  /home/doug/Source/blog/src/client/share_test.ts:4:24

    2 | import { makeShareable } from './share';
    3 |
  > 4 | const foo = globalThis.test;
      |                        ^^^^
    5 |
    6 | describe('share', () => {
    7 |     describe('makeShareable()', () => {

[TypeScript] Found 1 error. Watching for file changes. (x2)
 ✓ src/client/share_test.ts (2)

 Test Files  1 passed (1)
      Tests  2 passed (2)
Type Errors  no errors
   Start at  18:20:24
   Duration  1.31s (transform 0ms, setup 0ms, collect 15ms, tests 0ms, environment 0ms, prepare 316ms)

 PASS  Waiting for file changes...
       press h to show help, press q to quit
 ✓ src/client/share_test.ts (2)

There's some coloring which makes this a little more readable, but I'm particularly unhappy with the big "PASS" at the bottom which hides the TS error above it. If you don't see that, you'd never know there was a type issue. If a test fails, then the whole bottom of the terminal is dominated by a complex runtime error when most like the type issue is indicating the same problem in much more direct output. It feels like the TS check is just running alongside Vitest and printing to the same terminal, but otherwise is a completely disconnected process which does not influence test results. I wish it would at least fail the test execution and potentially even prevent the tests from running in the first place (that would probably have some performance impact, for a project of this size, I really don't care).

A small but really annoying nitpick I have is that "[TypeScript] Found 1 error. Watching for file changes." is displayed in the default text color but if there are 0 errors, the color is red, and makes me think something's broken on an otherwise passing run. I feel like the other way around would be more correct?

As I understand it, I believe the Vite community would generally recommend running tsc --watch --noEmit in a separate terminal, though I'm personally not a fan of the UX of managing multiple terminals or trying to run them both in parallel with some tooling to interleave the process output. Again, it's too easy to miss a type checking error and spend a lot of time debugging a runtime error the type system was trying to tell you about.

Beyond this, the checker plugin only runs on watch mode runs. Doing a single execution does not type check at all and AFAICT it is not possible to make vitest run fail on type check error. Instead you need to run tsc --noEmit && vitest run. That's not terrible, but does have a performance impact (which bugs me in theory, but not really in practice for a project this size) and forks my type checking logic between tsc for single run tests and vite-plugin-checker for watched executions.

Ultimately all these config issues trace back to the lack of a proper build system and my preferred UX not matching up with the Vite community. In fairness it is a problem with the existing Karma setup I have here, though it at least runs the type checker as part of the test. I seriously considered throwing all this away and instead doing a tsc | vite run approach by prebuilding my sources. Unfortunately for this Eleventy blog that's not super straightforward, but if I ever switch to @rules_prerender that might be more viable.

Playwright

I used Playwright as the browser driver (not sure why I really need to care what browser driver is used), but I found that installing Vitest/Playwright automatically installed Chrome, Firefox, and Webkit despite me explicitly choosing only Chrome in the setup options. This meant I needed to download another ~160MB of content for seemingly no reason. Playwright seems to use a separate command for installing browsers from npm install, which at least means this download pain shouldn't be too frequent. However that also means CI needs to manually install these browsers, which I'm not super happy with but is probably a better trade-off than auto installing in npm install.

Migration

You can't spy on module imports like my current Karma Jasmine set up can, meaning I need refactor a number of tests beyond just syntactic transformations.

import * as mod from './some-mod';
vi.spyOn(mod, 'func'); // ERROR

I suspect this will be an issue for any test runner with native ESM support, so it's probably a problem I'll have to deal with eventually. It's not a hard blocker here but it makes the migration a bit harder.

Conclusion

As much as I don't care for the browser UI and the general testing UX issues annoy me, I could pretty easily overlook them. The migration to Vitest isn't too painful for a project of this size. But the lack of debuggability is the real killer I can't look past right now.

I'm fairly confident I could build what I want in Web Test Runner but I'd rather not maintain my own Jasmine integration here and I'm not sold on Mocha as a feature-compatible replacement.

For now, I'm inclined to sit on Karma for the time being. It's still working fine in its current state, even if it is deprecated. The debuggability issue in Vitest browser mode will probably be fixed eventually and the browser UI will likely be improved. If so, I can revisit this and try again. I'm skeptical the testing UX with respect to TypeScript will change significantly, so I will likely need to learn to live with it or wait to migrate to @rules_prerender.