Stryker using all CPUs of the machine and ignoring concurrency configuration on Ubuntu 22.10.

[SmashingQuasar]

Summary

Hey! :wave:

I am running Stryker on Ubuntu 22.10 and for some reason it is using every single CPU available by default leading to crash of the OS. I tried adding the concurrency parameter to the configuration but to no avail.

I observed the usage through htop and it clearly shows that right when Stryker starts running unit tests, all CPUs are being used to their maximum without any room for any other process. The OS is able to survive in this situation as long as no mutant are timing out. When mutants start timing out, there is not enough CPU being freed for the OS to survive and it ultimately freezes and never comes back.

If that helps, my CPU is an Intel Core i9 9900k @3.6Ghz (8 pCores, 16 vCores).

Stryker config

{
    "packageManager": "pnpm",
    "mutate": ["./src/**/*.ts"],
    "ignorePatterns": [
        "**/node_modules/**",
        "**/.github/**",
        "**/.husky/**",
        "**/.stryker-tmp/**",
        "**/stats/**",
        "**/docs/**",
        "**/documentations/**",
        "**/reports/**",
        "**/coverage/**",
        "**/build/**",
        "**/dist/**",
        "**/lib/**"
    ],
    "concurrency": 6,
    "reporters": ["html", "clear-text", "progress"],
    "htmlReporter": {
        "fileName": "./reports/stryker/mutation.html"
    },
    "jsonReporter": {
        "fileName": "./reports/stryker/mutation.json"
    },
    "commandRunner": {
        "command": "pnpm test:unit:stryker"
    },
    "coverageAnalysis": "perTest",
    "ignoreStatic": true,
    "checkers": ["typescript"],
    "tsconfigFile": "tsconfig.stryker.json",
    "plugins": [
        "@stryker-mutator/mocha-runner",
        "@stryker-mutator/typescript-checker"
    ]
}

Test runner config

.mocharc.json:

{
    "$schema": "https://json.schemastore.org/mocharc",
    "extension": [
      "ts"
    ],
    "spec": "./__tests__/**/*.spec.ts",
    "require": ["./__tests__/__init__/setup.ts"],
    "loader": "ts-node/esm"
}

Stryker environment

pnpm-lock.yaml file regarding Stryker.

lockfileVersion: '6.0'

importers:

  .:
    devDependencies:
      '@stryker-mutator/core':
        specifier: ^6.4.2
        version: 6.4.2(typescript@5.0.4)
      '@stryker-mutator/mocha-runner':
        specifier: ^6.4.2
        version: 6.4.2(@stryker-mutator/core@6.4.2)(mocha@10.2.0)
      '@stryker-mutator/typescript-checker':
        specifier: ^6.4.2
        version: 6.4.2(@stryker-mutator/core@6.4.2)(typescript@5.0.4)

pnpm-lock.yaml file regarding Mocha and the gang:

lockfileVersion: '6.0'

importers:

  .:
    devDependencies:

      c8:
        specifier: ^7.13.0
        version: 7.13.0
      chai:
        specifier: ^4.3.7
        version: 4.3.7
      chai-as-promised:
        specifier: ^7.1.1
        version: 7.1.1(chai@4.3.7)
      mocha:
        specifier: ^10.2.0
        version: 10.2.0
      mochawesome:
        specifier: ^7.1.3
        version: 7.1.3(mocha@10.2.0)
      ts-node:
        specifier: ^10.9.1
        version: 10.9.1(@swc/core@1.3.56)(@types/node@20.1.0)(typescript@5.0.4)

Test runner environment

NODE_ENV=test TS_NODE_PROJECT=./tsconfig.stryker.json mocha --config .mocharc.stryker.json --parallel

tsconfig.stryker.json file:

{
    "extends": "./tsconfig.json",
    "compilerOptions": {
        "sourceMap": true
    },
    "include": [
        "./src/**/*",
        "./__tests__/**/*"
    ]
}

Local parent tsconfig.json:

{
    "$schema": "https://json.schemastore.org/tsconfig",
    "extends": "../../tsconfig.json",
    "include": [
        "./src",
        "./__tests__"
    ],
    "exclude": ["./node_modules"]
}

Mono-repository root tsconfig.json:

{
"$schema": "https://json.schemastore.org/tsconfig",
"compileOnSave": false,
"compilerOptions": {
    "sourceMap": true,
    "declaration": false,
    "moduleResolution": "NodeNext",
    "importHelpers": true,
    "noEmit": true,
    "pretty": true,
    "target": "ESNext",
    "module": "NodeNext",
    "lib": ["ESNext"],
    "skipLibCheck": true,
    "skipDefaultLibCheck": true,
    "newLine": "LF",
    "baseUrl": ".",
    "allowJs": false,
    "checkJs": false,
    "allowUnreachableCode": false,
    "allowUnusedLabels": false,
    "alwaysStrict": true,
    "exactOptionalPropertyTypes": true,
    "forceConsistentCasingInFileNames": true,
    "noErrorTruncation": true,
    "noFallthroughCasesInSwitch": true,
    "noImplicitAny": true,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noImplicitOverride": true,
    "noPropertyAccessFromIndexSignature": true,
    "noUncheckedIndexedAccess": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "strict": true,
    "strictBindCallApply": true,
    "strictFunctionTypes": true,
    "strictNullChecks": true,
    "strictPropertyInitialization": true,
    "useUnknownInCatchVariables": true,
    "preserveConstEnums": true,
    "noStrictGenericChecks": false,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": false
},
"exclude": ["node_modules"]
}

Your Environment

software	version(s)
node	18.7
npm	-
pnpm	8.4.0
Operating System	Ubuntu 22.10

Add stryker.log

It is not possible to generate a stryker.log file since the OS crashes when running the command.

[nicojs]

I think I see the problem here. You don't have a testRunner: 'mocha'. This means Stryker will use the command test runner, which runs npm test by default. Your normal npm test command is probably quite expensive, and running it in parallel is not great.

[SmashingQuasar]

Thanks a lot for your quick answer! (I'm always amazed how fast you answer to issue, it's just mind blowing. :heart: )

Following your advice, I read the documentation regarding using a built-in runner and I switched to stryker.conf.json to this:

{
    "packageManager": "pnpm",
    "mutate": ["./src/**/*.ts"],
    "ignorePatterns": [
        "**/node_modules/**",
        "**/.github/**",
        "**/.husky/**",
        "**/.stryker-tmp/**",
        "**/stats/**",
        "**/docs/**",
        "**/documentations/**",
        "**/reports/**",
        "**/coverage/**",
        "**/build/**",
        "**/dist/**",
        "**/lib/**"
    ],
    "concurrency": 6,
    "reporters": ["html", "clear-text", "progress"],
    "htmlReporter": {
        "fileName": "./reports/stryker/mutation.html"
    },
    "jsonReporter": {
        "fileName": "./reports/stryker/mutation.json"
    },
    "testRunner": "mocha",
    "coverageAnalysis": "perTest",
    "ignoreStatic": true,
    "checkers": ["typescript"],
    "tsconfigFile": "tsconfig.stryker.json",
    "plugins": [
        "@stryker-mutator/mocha-runner",
        "@stryker-mutator/typescript-checker"
    ],
    "mochaOptions": {
      "config": ".mocharc.stryker.json"
    }
}

Unfortunately it no longer works with TypeScript as I am getting the following error:

11:30:57 (18299) INFO ProjectReader Found 54 of 294 file(s) to be mutated.
11:30:57 (18299) INFO Instrumenter Instrumented 54 source file(s) with 558 mutant(s)
11:30:57 (18299) INFO ConcurrencyTokenProvider Creating 3 checker process(es) and 3 test runner process(es).
11:31:01 (18299) ERROR Stryker Unexpected error occurred while running Stryker StrykerError: SyntaxError: Unexpected token ')'
/home/smashing-quasar/development/projects/VitruviusLab/typescript/packages/strict-predicate/.stryker-tmp/sandbox9043502/__tests__/__init__/setup.ts:3
    log: (): void => {}
          ^

SyntaxError: Unexpected token ')'
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1187:20)
    at Module._compile (node:internal/modules/cjs/loader:1231:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1321:10)
    at Module.load (node:internal/modules/cjs/loader:1125:32)
    at Module._load (node:internal/modules/cjs/loader:965:12)
    at Module.require (node:internal/modules/cjs/loader:1149:19)
    at require (node:internal/modules/helpers:121:18)
    at exports.requireOrImport (/home/smashing-quasar/development/projects/VitruviusLab/typescript/node_modules/.pnpm/mocha@10.2.0/node_modules/mocha/lib/nodejs/esm-utils.js:53:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ChildProcess.<anonymous> (file:///home/smashing-quasar/development/projects/VitruviusLab/typescript/node_modules/.pnpm/@stryker-mutator+core@6.4.2_typescript@5.0.4/node_modules/@stryker-mutator/core/dist/src/child-proxy/child-process-proxy.js:149:68)
    at ChildProcess.emit (node:events:511:28)
    at emit (node:internal/child_process:944:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
  innerError: undefined
}

[SmashingQuasar]

I tried inverting the plugins section order to put @stryker-mutator/typescript-checker before @stryker-mutator/mocharunner in case the order was important but it did not change anything.

I also attempted to edit the mochaOptions section so my stryker.conf.json now looks like this:

{
    "packageManager": "pnpm",
    "mutate": ["./src/**/*.ts"],
    "ignorePatterns": [
        "**/node_modules/**",
        "**/.github/**",
        "**/.husky/**",
        "**/.stryker-tmp/**",
        "**/stats/**",
        "**/docs/**",
        "**/documentations/**",
        "**/reports/**",
        "**/coverage/**",
        "**/build/**",
        "**/dist/**",
        "**/lib/**"
    ],
    "concurrency": 6,
    "reporters": ["html", "clear-text", "progress"],
    "htmlReporter": {
        "fileName": "./reports/stryker/mutation.html"
    },
    "jsonReporter": {
        "fileName": "./reports/stryker/mutation.json"
    },
    "testRunner": "mocha",
    "coverageAnalysis": "perTest",
    "ignoreStatic": true,
    "checkers": ["typescript"],
    "tsconfigFile": "tsconfig.stryker.json",
    "plugins": [
        "@stryker-mutator/typescript-checker",
        "@stryker-mutator/mocha-runner"
    ],
    "mochaOptions": {
        "extension": [
            "ts"
        ],
        "spec": ["./__tests__/**/*.spec.ts"],
        "config": ".mocharc.stryker.json",
        "require": ["./__tests__/__init__/setup.ts"]
    },
    "typescriptChecker": {
      "prioritizePerformanceOverAccuracy": false
    }
}

[SmashingQuasar]

I also want to point out that it used to work just fine with commandRunner before, it only recently started to act up and I am unsure why. :thinking:

[SmashingQuasar]

So I figured out why all CPUs were being used with no mercy.

My command was: NODE_ENV=test TS_NODE_PROJECT=./tsconfig.stryker.json mocha --config .mocharc.stryker.json --parallel

The --parallel for Mocha created this problem because it was attempting to parallelise on top of Stryker.

However I am very interested in making Stryker work with the built-in mocha runner and TypeScript if you have time to point my error @nicojs. :pray:

[nicojs]

Hi, sorry it took me a while to get back to you.

Indeed, I overlooked the --parallel at first. Good you've found it! 🎉

I'm pretty sure your test command isn't the full story:

NODE_ENV=test TS_NODE_PROJECT=./tsconfig.stryker.json mocha --config .mocharc.stryker.json --parallel

I've noticed you're using a loader option in your .mocharc.stryker.json file, but I've never heard of this option. However, it is pointing heavily in the direction that you're using ts-node/esm to transpile your files on the fly using something like node --loader ts-node/esm.

If you want to load a loader with Stryker, you should pass it as a node arg like this:

{
  "testRunnerNodeArgs": ["--loader", "ts-node/esm"]
}

See https://stryker-mutator.io/docs/stryker-js/configuration/#testrunnernodeargs-string

Since ts-node is type checking on the fly and Stryker makes compile errors (no way around that), you should probably also configure disableTypeChecks:

{
  "disableTypeChecks": true
}

Note: this is unrelated to the @stryker-mutator/typescript-checker, you can still use that to filter-out mutants that create compile errors

Instead of disableTypeChecks, you can also use --transpileOnly

[SmashingQuasar]

Hey, no worries for the delay, all good! Thanks a lot for your answer! :heart:

I've noticed you're using a loader option in your .mocharc.stryker.json file, but I've never heard of this option.

Any CLI option for Mocha can be put within the .mocharc.json file. We just find it cleaner to store it within this file rather than passing it directly to the runner. It's the same as mocha --loader=ts-node/esm. I would like to use stc (tsc redone with Rust) instead but I don't think it catches all typing errors just yet and we may end up with a lot of false positive.

I do have one more question though.

Since Node.JS recently released (18+) their built-in test runner (and because it is blazing fast), we are considering moving to this test runner instead. We've made it work successfully, however it cannot be ran sequentially. The command node --test will run in parallel mode no matter what. Obviously this creates the same issue as the original one that I had (hence why I am mentioning it here instead of a new issue). I don't know if you have already imagined a workaround for this.

[nicojs]

The node --test test runner is using node-tap under the hood. We are working on a @stryker-mutator/tap-runner that supports this as well.

One of the design goals of node-tap is to be able to simply import a test file directly. This is what the @stryker-mutator/tap-runner does. This way we control the concurrency. This does mean however that any --loader functionality would have to be reconfigured using testRunnerNodeArgs.

However, before you switch, do some tests yourself to verify performance. The way I understand it is that node-tap is much slower than mocha. Node-tap always creates a new process for each file. This is the way if sandboxes test runs. This is also a trend we notice in other test runners, Vitest and jest also do this and Stryker tries and usually succeeds in disabling it.

As for Stryker, the @stryker-mutator/tap-runner will be released within a month. It will be much slower than @stryker-mutator/mocha-runner because it also has to create a process per test file per run. Coverage analysis will work, but is not fine-grained: it can only measure mutant coverage per test file, rather than per test within that file. The @stryker-mutator/mocha-runner however, supports fine-grained coverage and hot reload (reuse of processes). This makes the @stryker-mutator/mocha-runner probably the best choice for Stryker.

[SmashingQuasar]

Ok, thanks a lot for your answer! Looking forward to your implementation of node-tap! :+1:

We just did a benchmark for our project and the native runner was 60 times (yes, really) faster than Mocha. An important thing to note is that we are completely dependency-free so it may change things. I wonder why we so much of a difference though, maybe this is hardware related? Have you done your benchmark on arm64 or amd64? May I ask what kind of CPU was running the tests?

Edit: Fixed a typo, there was an extra 0.

[nicojs]

Btw, node-tap support got released last week: https://stryker-mutator.io/blog/announcing-stryker-js-7/ 🎉

[SmashingQuasar]

Thanks a lot for the update! I'm going to have a look at this when I can, unfortunately I am under heavy workload these days so it won't be before a few weeks. I'll keep you posted about it! 👍