Closed mattzeunert closed 9 years ago
Hi Matt,
Sorry for not getting back to you earlier. Both Martin and I have been away quite a lot, and have been focusing mainly on splitting the project, rather than looking into the current one.
tl;dr I've fixed your issue in #44, hoping to release a new version (1.2.3) this weekend. See below for a detailed description of the problem.
It took me quite a while to figure out what exactly was going wrong, but I finally managed to find the problem. As you found out, the program basically goes through three phases, in which it is serving:
[ 'src/add.js', 'test/*.js' ]
[ 'src/add.js', 'test/add.spec.js' ]
[ 'src/add.js' ]
This, however, is not wat it should be doing. In the first phase, it should do one run with just the code files, and no specs (in order to establish a base coverage). Phase two consists of running every spec file one by one against all code files (in order to find out which spec file covers which code file(s)). In the third and final phase, all code files should be served one by one, along with the spec files that cover them (as has been establised in phase 1 and 2). In your case, the list should therefore be:
[ 'src/add.js']
[ 'src/add.js', 'test/add.spec.js' ]
[ 'src/add.js', 'test/add.spec.js' ]
Now, why was this happening? Turns out we are merging the list of files to serve with the user-provided Karma configuration, rather than overwriting it. I.e., you configured
var karmaConfig = {
//...
files: ['src/*.js', 'test/*.js'],
//...
};
which is then merged with ['src/*.js']
to, again, ['src/*.js', 'test/*.js']
.
The reason why this results in the "tests fail without mutation" error is simply that Karma marks complete absense of tests as an unsuccessful run. Coincidentally, I've also improved handling of this situation today (#42, to solve #41). It is now rightfully reported as "Could not properly set up Karma".
That brings me to your second issue. This appears to have been a side effect of hard-coding self._config.files
, as I was unable to reproduce it with the proper fix. It now rightfully outputs:
(14:10:46.926) INFO [mutation-testing]: Mutating file: [...]\mutation-testing11584-5232-adjg4o\src\add.js
(14:10:46.927) INFO [mutation-testing]: Mutating line 1, 1/3 (33%)
(14:10:47.094) INFO [mutation-testing]: Mutating line 2, 2/3 (67%)
(14:10:47.231) INFO [mutation-testing]: Mutating line 2, 3/3 (100%)
/src/add.js:2:11 Replaced + with - -> SURVIVED
(14:10:47.392) INFO [mutation-testing]: Done mutating file: [...]\mutation-testing11584-5232-adjg4o\src\add.js
2 of 3 unignored mutations are tested (66%).
To still answer your question: It is not really possible to see exactly what the code looks like after a mutation, but you can see exactly which mutations are performed when you configure the HTML reporter, e.g.:
{
// ...
mutationTest: {
options: {
reporters: {
html: {
dir: 'reports/mutation-test-html'
}
},
maxReportedMutationLength: 0 // Unlimited length of what has been mutated
},
// ...
}
// ...
}
Now, back to your issue. I have created pull request #44 for Martin to have a look at (we don't release anything without it being reviewed by the other). Hoping to release version 1.2.3 some time this weekend.
-- Jimi
PS: Thanks to your project, I also found out that the 'ALL' log level was not allowed for Karma. This is now fixed as well :-).
Update: I have just now released v1.2.3, in which this issue should be fixed.
I've been trying to get the plugin to work, but wasn't successful. Here's my code: https://github.com/mattzeunert/grunt-mutation-testing-example
There are two problems I'm running into: 1) Getting "Tests fail without mutations" message 2) Says survived after removing function, but if do it manually in Karma it fails as expected
I'm getting the first issue with the setup as it is in my repo.
Here's the output I'm getting, after adding a console.log for the config that is passed to Karma: https://gist.github.com/mattzeunert/bc338a18a97d86fd0da6
It seems to pass in these files:
Is that expected? Why am I getting the "tests fail without mutation" even though they work in Karma?
I've worked around the first issue by forcing
self._config.files = [ 'src/add.js', 'tests/add.spec.js' ];
in KarmaServerManager.This is the output I get after that: https://gist.github.com/mattzeunert/3f8dcf666eaa2238a9a5
When I remove the function myself the test doesn't survive in Karma. Is it possible to see exactly what the code looked like after the mutation?