How do you actually support your own test framework?

[NullVoxPopuli]

Seeing the docs here: https://modern-web.dev/docs/test-runner/test-frameworks/write-your-own/

I still have questions:

• how does the CLI communicate with a session?

• I have https://github.com/remcovaes/web-test-runner-vite-plugin with the following config

  // web-test-runner.config.mjs
  import { vitePlugin } from '@remcovaes/web-test-runner-vite-plugin';
  
  export default {
  plugins: [vitePlugin()],
  };

  but when I run web-test-runner, nothing happens.. the terminal is just.. blank?

• do the APIs from the test-frameworks/write-your-own page provide reporting? / https://modern-web.dev/docs/test-runner/reporters/overview/
• how does it know which browser to launch?

[thepassle]

A testing framework in this context means the client side testing library (for example, mocha which is the default, or jasmine) that will execute your tests and collect the results.

As for why the vitePlugin isn't giving any logs, that seems like an issue for the vite plugin, we don't maintain it. Maybe they disable some logs.

The browser launcher is separate from the test framework (e.g. mocha), and is not necessarily related to the testing framework.

[NullVoxPopuli]

A testing framework in this context means the client side testing library

yes, I'm using QUnit.

As for why the vitePlugin isn't giving any logs, that seems like an issue for the vite plugin, we don't maintain it. Maybe they disable some logs.

cool, I'll check over there, thanks

The browser launcher is separate from the test framework

aye. I see here: https://modern-web.dev/docs/test-runner/browser-launchers/overview/ That puppeteer and Playwright are the only local-launchers that support many browsers... yet they download another copy of the browser during installation? this seems insane.

Is there a launcher that will just use what is locally installed? (testem does this, for example, very nice / fast installation, as it looks up what is on your system, rather than downloads extra stuff)

Though maybe webdriver would be the best option? I'm not sure what path is in these docs: https://modern-web.dev/docs/test-runner/browser-launchers/webdriver/

[NullVoxPopuli]

As for why the vitePlugin isn't giving any logs

even after removing the vite plugin, there is no output.

pnpm web-test-runner --config ./web-test-runner.config.mjs --debug --manual

and it just.. hangs?

[NullVoxPopuli]

oh this is interested, I updated files to be an empty array, because @web/test-runner is not responsible fro choosing what files to test -- vite is. But, when I did that, I finally got output from the CLI!

❯ pnpm web-test-runner --config ./web-test-runner.config.mjs --manual

Error: Did not find any tests to run. Use the "files" or "groups" option to configure test files.

Is there a way to skip this check? @web/test-runner has no business being concerned with test-files (unless it's to refresh the page in --manual mode? idk)

[NullVoxPopuli]

Got a reproduction here if anyone's curious: https://github.com/glimmerjs/glimmer-vm/pull/1424

[NullVoxPopuli]

Got a reproduction here if anyone's curious: https://github.com/glimmerjs/glimmer-vm/pull/1424

[NullVoxPopuli]

I walked away from my computer, leaving web-test-runner running, and came back to a heap limit issue:

<--- Last few GCs --->

[86414:0x118008000]   206414 ms: Mark-sweep 4058.0 (4141.4) -> 4044.3 (4143.6) MB, 350.7 / 0.0 ms  (average mu = 0.525, current mu = 0.397) allocation failure scavenge might not succeed
[86414:0x118008000]   207514 ms: Mark-sweep 4060.4 (4143.6) -> 4046.7 (4145.9) MB, 955.3 / 0.0 ms  (average mu = 0.298, current mu = 0.132) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1040df52c node::Abort() [<path/to/>/node/16.20.0/bin/node]

[NullVoxPopuli]

so.. that memory limit thing occurred because I had ['**/*.ts'] set as files -- which I removed in place of a "dummy.js" file -- which has nothing -- would be great to just omit the files config option.

now the webdriver plugin is connecting on the wrong port -- 4444 instead of 5173 🤔 I have output!!! yay!!!

[NullVoxPopuli]

I'm starting to think what I want to do isn't possible? I can't find any way to:

• have the test-runner wait for a server to boot (vite, in this case)
• have the test-runner connect to a URL on that server (the tests html, in this case)

[thepassle]

oh this is interested, I updated files to be an empty array, because @web/test-runner is not responsible fro choosing what files to test -- vite is.

@web/test-runner is definitely responsible for choosing the test files, not vite. Vite just has to serve them.

[NullVoxPopuli]

I don't want that behavior, as i have vite already set up.

[thepassle]

I don't want that behavior, as i have vite already set up.

Vite is a dev server(/build tool). Wtr is a test runner. They are different tools. The test runner needs a dev server to resolve your source code, but the test runner is responsible for figuring out which tests should be ran.

[NullVoxPopuli]

A communication protocol could be established between the test runner, and the dev server -- but maybe your on to something. Maybe i don't need a test runner. I need a test.. launcher?