Open michahell opened 9 months ago
Hi @michahell thanks for opening this issue 🙋♂️. Also thanks for sharing your build server logs, it is crucial in figuring out what is wrong.
I'm pretty sure the cause here is #2448, an older issue that sometimes seems to pop up.
Stryker uses tree-kill to kill worker processes (checkers, test runners) when they're done. This ensures no dangling browsers, web servers, etc are running and accumulating during mutation testing (for example, killing karma
might cause your chrome browser to keep running on linux-based systems). However, in rare circumstances, it seems that dependencies of tree-kill
are not installed. For example, ps
seems to be missing on your build server.
I've created a small test project to verify that tree-kill
works (or indeed, doesn't work). Could you try to run it on your build server? Cloning it and running npm start
should do the trick.
Hello @nicojs !
Thanks for your elaborate response and explanation. For speed (not having to create a separate repo + build pipeline) I've included tree-kill
as an explicit dependency in our project and made my branch only execute the node script ISO stryker, this is the result locally on my machine:
❯ node stryker/tree-kill-test.mjs
child process exited with code 1, signal null
Child process killed
and this in our pipeline:
$ node stryker/tree-kill-test.mjs
child process exited with code 1, signal null
node:events:495
throw er; // Unhandled 'error' event
^
Error: spawn ps ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:284:19)
at onErrorNT (node:internal/child_process:477:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:290:12)
at onErrorNT (node:internal/child_process:477:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn ps',
path: 'ps',
spawnargs: [ '-o', 'pid', '--no-headers', '--ppid', 27 ]
}
Node.js v18.18.2
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
It seems like this is indeed the culprit.
I'm going to try to install the ps
command in the build base-image that we're using with apt-get update
+ apt-get install procps
and see if that remedies it.
will report back after!
Thanks, I hope that works. Looking through the tree-kill code it seems that ps
is the only dependency for linux.
I'll also re-open #2448, since that is the cause. I think the best thing to do when tree-kill
fails is to perform a 'normal' childProcess.kill
.
after installing procps
on our build image, the tree-kill
test code now does the same as on my local machine:
$ node stryker/tree-kill-test.mjs
child process exited with code 1, signal null
Child process killed
I'll try running Stryker-mutator again to see if --maxTestRunnerReuse 20
is going to save us :)
stryker-mutator seems to now keep chugging on in our pipeline!
We now have a working baseline, we can increment --maxTestRunnerReuse 20
to see when memory pressure becomes too large 👍
Summary
We are getting an
Unhandled 'error' event
Node.js error when running Stryker-mutator in our GitLab pipeline.We are adding stryker-mutator to our Angular 16 frontend repo, size of about 500k sloc.
Locally on my M1 macbook I can easily run stryker without memory or unhandled issues, which I did, to build up an incremental file of about 60MB (~31.000 mutants) by running 'subsets' of folders through stryker. This is how we do that:
and using:
concurrency:2
I can see that the test runner child process can spike to ~3,5GB memory and also the stryker process itself can spike to ~1,5GB.stryker.log
as a pipeline artefact, but that is something that I can change.Stryker config
from build log:
Test runner config
karma.conf.cjs
Stryker environment
Test runner environment
karma:
and jasmine:
test command
Your Environment
Add stryker.log
This I don't have unfortunately, since stryker is being run in our pipeline and stryker.log is currently not an artifact. I can make it so and update this issue in the future