Exporting native runner coverage with TypeScript

[SmashingQuasar]

Question

Hey! Sorry if this is not the right place to ask but I am going through some strange issues with the Node native test runner. I am a long time user of Stryker (which I consider a must-use in every project).

We have been using the native runner with Stryker within two of our packages: ts-predicate and mockingbird.

Those two repositories are also using Stryker JS. Somehow, Stryker is able to extract to real coverage outputted by the Node native runner whilst still working with TypeScript. I never managed to get the coverage to work with the native runner. It always tells me all of my files have only 1 line of code and the coverage is 100% on every file (outside of Stryker, when I am running my unit tests).

My question is: What black magic did you use to get a proper coverage report from the native runner whilst using JIT compilation for TypeScript? 😁

Stryker environment

├── @stryker-mutator/core@8.2.3 -> ./node_modules/.pnpm/@stryker-mutator+core@8.2.3/node_modules/@stryker-mutator/core
├── @stryker-mutator/typescript-checker@8.2.3 -> ./node_modules/.pnpm/@stryker-mutator+typescript-checker@8.2.3_@stryker-mutator+core@8.2.3_typescript@5.1.6/node_modules/@stryker-mutator/typescript-checker

I am using the native test runner for Node.js in version 20.11.1. I am using the native assertion library as well. I was trying to use c8 to retrieve the coverage.

Additional context

If you don't know out of the box, feel free to close this question as it is not directly tied to the Stryker behaviour. I just tried my luck here, I don't want to bother you. :)

[nicojs]

Somehow, Stryker is able to extract to real coverage outputted by the Node native runner whilst still working with TypeScript.

Do you mean inside the StrykerJS mutation testing report? So the mutant coverage? That shouldn't be. If you're using the command test runner. However, it does work if you're using the tap runner

What are you trying to do exactly? Are you trying to create a code coverage report for your project? Or are you trying to configure StrykerJS correctly?

[SmashingQuasar]

Thanks for your answer!

I'm trying to reverse-engineer Stryker to understand why Stryker is able to calculate coverage and why it does not work correctly with the native runner on my project outside of Stryker.

Basically, Stryker is properly understanding the coverage en works great but I can't get a proper coverage report with any tool (like c8) despite trying to use TAP.

I know it's a long shot. It's the only thing that I need to use the native runner and abandon Mocha so I'm trying what I can. ;-/

[nicojs]

What does your Stryker config look like?

[SmashingQuasar]

This is the Stryker configuration file:

{
    "packageManager": "pnpm",
    "mutate": ["./src/**/*.mts"],
    "ignorePatterns": [
        "**/node_modules/**",
        "**/.github/**",
        "**/.husky/**",
        "**/.stryker-tmp/**",
        "**/stats/**",
        "**/docs/**",
        "**/documentations/**",
        "**/reports/**",
        "**/coverage/**",
        "**/build/**",
        "**/dist/**",
        "**/lib/**"
    ],
    "concurrency": 6,
    "reporters": ["html", "clear-text", "progress"],
    "htmlReporter": {
        "fileName": "./reports/stryker/mutation.html"
    },
    "jsonReporter": {
        "fileName": "./reports/stryker/mutation.json"
    },
    "commandRunner": {
        "command": "pnpm test:unit:stryker"
    },
    "coverageAnalysis": "perTest",
    "ignoreStatic": true,
    "checkers": ["typescript"],
    "tsconfigFile": "tsconfig.stryker.json",
    "plugins": [
        "@stryker-mutator/typescript-checker"
    ],
    "typescriptChecker": {
      "prioritizePerformanceOverAccuracy": false
    }
}

[nicojs]

Basically, Stryker is properly understanding the coverage en works great [...] This is the Stryker configuration file: [...]

This is a mistake. Stryker doesn't understand the coverage. You've configured the command test runner. This runner doesn't measure code coverage, as it has too little information. This results in Stryker running every test for every mutant (by running your configured command). If the command exits with a non-0 exit code, it assumes the tests failed, and the mutant is reported as "Killed". If it exits with a 0 exit code, it assumes the tests passed, and the mutant is reported as "Survived". This results in a mutation score (sometimes called mutant coverage). This may be what you mean by 'code coverage'.

You've configured perTest coverage analysis, but that doesn't affect the command test runner.

If you're using the native test runner, as in node --test, then you should be able to use the TAP test runner plugin. It supports per-test coverage analysis, reporting each test file as a test. It should make your Stryker run much faster, and your mutation report more detailed, as it also shows which test covers which mutant.

[SmashingQuasar]

Thank for your answer and your help. It seems I didn't know really understand how Stryker works. I'll ask the author or c8 to see if he knows something about that.

Thanks for the precision about the TAP test runner plugin. This will be of great help for some of our projects within VitruviusLabs.

I'll close this issue since it was already quite borderline related to Stryker. Sorry for the inconvenience and thanks again! ;)

[nicojs]

We also have c8 setup for regular code coverage on StrykerJS. It works for us, so maybe you can also take a look there if you're interested.

Example config for the core package: https://github.com/stryker-mutator/stryker-js/blob/master/packages/core/.nycrc.json