How to set it up correctly

[mattzeunert]

I've been trying to get the plugin to work, but wasn't successful. Here's my code: https://github.com/mattzeunert/grunt-mutation-testing-example

There are two problems I'm running into: 1) Getting "Tests fail without mutations" message 2) Says survived after removing function, but if do it manually in Karma it fails as expected

I'm getting the first issue with the setup as it is in my repo.

Here's the output I'm getting, after adding a console.log for the config that is passed to Karma: https://gist.github.com/mattzeunert/bc338a18a97d86fd0da6

It seems to pass in these files:

1. [ 'src/add.js', 'test/*.js' ]
2. [ 'src/add.js', 'test/add.spec.js' ]
3. [ 'src/add.js' ]

Is that expected? Why am I getting the "tests fail without mutation" even though they work in Karma?

I've worked around the first issue by forcing self._config.files = [ 'src/add.js', 'tests/add.spec.js' ]; in KarmaServerManager.

This is the output I get after that: https://gist.github.com/mattzeunert/3f8dcf666eaa2238a9a5

(13:40:31.573) INFO [mutation-testing]: Mutating file:     /var/folders/tg/r1vq4wwd641fbx971h2gyqp00000gn/T/mutation-testing11579-30414-h3y7d7/src/add.js
(13:40:31.575) INFO [mutation-testing]: Mutating line 1, 1/3 (33%)
(13:40:31.575) TRACE [KarmaServer]: Server status changed to: RUNNING
(13:40:31.710) TRACE [KarmaServer]: Server status changed to: READY

/src/add.js:1:1 Removed function add(a, b){ return a + b; } -> SURVIVED

When I remove the function myself the test doesn't survive in Karma. Is it possible to see exactly what the code looked like after the mutation?

[jimivdw]

Hi Matt,

Sorry for not getting back to you earlier. Both Martin and I have been away quite a lot, and have been focusing mainly on splitting the project, rather than looking into the current one.

tl;dr I've fixed your issue in #44, hoping to release a new version (1.2.3) this weekend. See below for a detailed description of the problem.

---

It took me quite a while to figure out what exactly was going wrong, but I finally managed to find the problem. As you found out, the program basically goes through three phases, in which it is serving:

1. [ 'src/add.js', 'test/*.js' ]
2. [ 'src/add.js', 'test/add.spec.js' ]
3. [ 'src/add.js' ]

This, however, is not wat it should be doing. In the first phase, it should do one run with just the code files, and no specs (in order to establish a base coverage). Phase two consists of running every spec file one by one against all code files (in order to find out which spec file covers which code file(s)). In the third and final phase, all code files should be served one by one, along with the spec files that cover them (as has been establised in phase 1 and 2). In your case, the list should therefore be:

1. [ 'src/add.js']
2. [ 'src/add.js', 'test/add.spec.js' ]
3. [ 'src/add.js', 'test/add.spec.js' ]

Now, why was this happening? Turns out we are merging the list of files to serve with the user-provided Karma configuration, rather than overwriting it. I.e., you configured

var karmaConfig = {
  //...
  files: ['src/*.js', 'test/*.js'],
  //...
};

which is then merged with ['src/*.js'] to, again, ['src/*.js', 'test/*.js'].

The reason why this results in the "tests fail without mutation" error is simply that Karma marks complete absense of tests as an unsuccessful run. Coincidentally, I've also improved handling of this situation today (#42, to solve #41). It is now rightfully reported as "Could not properly set up Karma".

That brings me to your second issue. This appears to have been a side effect of hard-coding self._config.files, as I was unable to reproduce it with the proper fix. It now rightfully outputs:

(14:10:46.926) INFO [mutation-testing]: Mutating file: [...]\mutation-testing11584-5232-adjg4o\src\add.js
(14:10:46.927) INFO [mutation-testing]: Mutating line 1, 1/3 (33%)
(14:10:47.094) INFO [mutation-testing]: Mutating line 2, 2/3 (67%)
(14:10:47.231) INFO [mutation-testing]: Mutating line 2, 3/3 (100%)

/src/add.js:2:11 Replaced  +  with - -> SURVIVED
(14:10:47.392) INFO [mutation-testing]: Done mutating file: [...]\mutation-testing11584-5232-adjg4o\src\add.js

2 of 3 unignored mutations are tested (66%).

To still answer your question: It is not really possible to see exactly what the code looks like after a mutation, but you can see exactly which mutations are performed when you configure the HTML reporter, e.g.:

{
  // ...
  mutationTest: {
    options: {
      reporters: {
        html: {
          dir: 'reports/mutation-test-html'
        }
      },
      maxReportedMutationLength: 0 // Unlimited length of what has been mutated
    },
    // ...
  }
  // ...
}

Now, back to your issue. I have created pull request #44 for Martin to have a look at (we don't release anything without it being reviewed by the other). Hoping to release version 1.2.3 some time this weekend.

-- Jimi

PS: Thanks to your project, I also found out that the 'ALL' log level was not allowed for Karma. This is now fixed as well :-).

[jimivdw]

Update: I have just now released v1.2.3, in which this issue should be fixed.