JavaScript Mutation Testing as grunt plugin. Tests your tests by mutating the code.
Note
We will be working on (gradually) migrating the majority of the code base to the Stryker project.
For now, Stryker only supports Jasmine tests, but support for other frameworks will be added in the future. If you are currently using grunt-mutation-testing to assess your Jasmine tests, or planning to do so, please consider trying out Stryker. We would be happy to receive some feedback, either through GitHub issues or on Gitter.
This plugin requires Grunt.
If you haven't used Grunt before, be sure to check out the Getting Started guide, as it explains how to create a Gruntfile as well as install and use Grunt plugins. Once you're familiar with that process, you may install this plugin with this command:
npm install grunt-mutation-testing --save-dev
Once the plugin has been installed, it may be enabled inside your Gruntfile with this line of JavaScript:
grunt.loadNpmTasks('grunt-mutation-testing');
In your project's Gruntfile, add a section named mutationTest
to the data object passed into grunt.initConfig()
.
grunt.initConfig({
mutationTest: {
options: {
// Task-specific options go here.
},
your_target: {
// Target-specific options go here.
},
},
})
Required
Type: String
or [String]
List of source code files needed to successfully run your unit tests (including required libraries).
Required
Type: String
or [String]
List of unit test specifications that need to be run.
Required
Type: String
or [String]
List of source code files that should be mutation tested.
optional
Type: String
Default: "."
Base path from which to look for the code, specs, and mutation files
optional
Type: String
Default: "karma"
The test framework to use. Available values: karma
, mocha
.
optional
Type: Object
Karma-specific options. See the Karma documentation for available options.
In order for Karma to be able to run properly, it is necessary to at least provide the frameworks
and browsers
options. You should usually be able to reuse your existing Karma config file for this (by supplying it in the configFile
option).
Note that some options (basePath
, files
, background
, singleRun
, and autoWatch
) are overwritten automatically by the mutation testing framework.
optional
Type: Number
Default: 5
Maximum number of Karma servers that can be active at the same time. They run on ports subsequent to the configured Karma port, e.g. by default, Karma servers can be started on ports: 12111
, 12112
, 12113
, 12114
, and 12115
; the next server will then be started on port 12111
again.
optional
Type: Number
Default: 10
Maximum time (in seconds) to wait for a Karma server to start up.
optional
Type: Number
Default: 2
Maximum time (in seconds) a single test run may take. Used to detect if the mutated code does not end up in an inifinite loop, but may trigger false positives because of this. If, under normal circumstances, your tests already take around 2 seconds to run, you should increase this property accordingly.
optional
Type: Object
Mocha-specific options. See the Mocha documentation for available options.
optional
Type: String
Default: INFO
The used log level. Available options: ALL
, TRACE
, DEBUG
, INFO
, WARN
, ERROR
, FATAL
, OFF
.
optional
Type: Object
Default: { console: true }
Configuration of reporters to use. Available options: console
, text
, html
, json
.
optional
Type: String
Default: "reports/grunt-mutation-testing"
Directory to place the text report in.
optional
Type: String
Default: grunt-mutation-testing.txt
Filename of the text report.
optional
Type: String
Default: "reports/grunt-mutation-testing/html"
Directory to place the HTML report in.
optional
Type: Number
Default: 80
Percentage of mutations that should be killed in order for a test result to be considered successful.
optional
Type: String
Default: "reports/grunt-mutation-testing/json"
Directory to place the JSON report in.
optional
Type: String
Default: mutations.json
Filename of the JSON report.
optional
Type: Number
Default: 80
The maximum reported length of the mutation that has been done. When set to 0
, the full mutation is logged regardless of its length.
optional
Type: String
or RegExp
or [String and/or RegExp]
Default: /('use strict'|"use strict");/
Code that matches with any of the supplied regular expressions will not be mutated in any way.
Note that, by default, mutations on the strict mode keyword 'use strict'
will be ignored. If you really do want to mutate it, this can be done by providing the options.discardDefaultIgnore
option (see below).
optional
Type: String
or RegExp
or [String and/or RegExp]
Mutation replacements that match with any of the supplied regular expressions will not be introduced.
optional
Type: Object
A set of properties, indicating whether certain mutations should be excluded for all files. See below for a list of available mutations.
optional
Type: Boolean
Default: false
When true, code is not copied to a temporary directory and mutated there, but instead the original production code is mutated, which can speed up your tests.
Be careful when using this option, as, in case the mutation process does not exit correctly, your code will be left mutated.
optional
Type: Boolean
Default: false
When true, mutations that are ignored by default (see options.ignore
, above) will no longer be ignored.
We do not really see any relevant use case for this, but did not want to make it impossible to perform certain mutations either. Hence the existence of this configuration option.
optional
Type: String
or Function
This test is executed for every Mutation. If it passes, this mutation is reported as 'survived'.
In this example, the default options are used to report every possible mutation.
grunt.initConfig({
mutationTest: {
options: {},
target: {
code: ['src/*.js', 'src/lib/myLib/*.js'],
specs: 'test/**/*KarmaSpec.js',
mutate: 'src/*.js'
}
}
});
In this example all mutations are reported, which cause no failure of the grunt script.
grunt.initConfig({
mutationTest: {
options: {
test: 'grunt test'
},
target: {
code: ['src/*.js', 'src/lib/myLib/*.js'],
specs: 'test/**/*KarmaSpec.js',
mutate: 'src/*.js'
}
}
});
Calling a test in this way is easy but very slow. It's much faster to call tests directly by providing a test function. This is demonstrated in this project's Gruntfile.js.
For your convenience you can easily configure fast mocha and karma tests:
grunt.initConfig({
mutationTest: {
options: {
testFramework: 'mocha'
},
target: {
code: ['src/*.js', 'src/lib/myLib/*.js'],
specs: 'test/**/*MochaSpec.js',
mutate: 'src/*.js'
}
}
});
grunt.initConfig({
mutationTest: {
options: {
karma: {
waitForServerTime: 10 // optional, only used for illustration purposes here
}
},
target: {
code: ['src/*.js', 'src/lib/myLib/*.js'],
specs: 'test/**/*KarmaSpec.js',
mutate: 'src/*.js'
}
}
});
In this section, you can find an overview of the most common errors you may encounter.
When you get the 'Tests fail without mutations [...]' error, it is most likely that something is wrong with your configuration. Usually, this means that something is wrong in the code
, specs
, mutate
part. You should check if code
really points to all your source files and the libraries you need, and if the paths to specs
and mutate
follow the correct base path. For more information on these options, check the Options section above.
Whereas usually, this means a mutation caused your code to enter an infinite loop (from which it cannot recover), it can also mean that the default timeout setting is too short for your tests to finish. You can test this by modifying the options.karma.waitForRunnerTime
config property, by which you set the maximum duration of a single test run (in seconds).
Currently, the following mutations are available:
Mutation code | Description | Example |
---|---|---|
MATH |
Replace arithmetic operators by their opposites | 1 + 1 to 1 - 1 |
ARRAY |
Remove elements from an array | [1,2,3] to [1,3] |
BLOCK_STATEMENT |
Remove statements from a block of statements | function foo(x) { x = x * 2; return x; } to function foo(x) { return x; } |
METHOD_CALL |
Mutate parameters of a function call | foo(x) to x |
COMPARISON |
Replace operators by their boundary and negation counterparts | x < 10 to x <= 10 |
LITERAL |
Replace strings, increment numbers, and negate booleans | var x = 'Hello' to var x = '"MUTATION!"' |
LOGICAL_EXPRESSION |
Replace logical operators by their opposites | x && y to x || y |
OBJECT |
Remove object properties | {a: 10, b: 'B'} to {b: 'B'} |
UNARY_EXPRESSION |
Negate unary expressions | var x = -42 to var x = 42 |
UPDATE_EXPRESSION |
Negate update expressions | x++ to x-- |
Since not all mutations may be relevant for your project, it is possible to configure which mutations should be performed and which should not.
In order to completely disable a specific mutation, one can provide the excludeMutations configuration option. This takes an object where the keys represent the mutations and the values denote whether the mutation should in fact be excluded. For example:
{
mutationTest: {
options: {
excludeMutations: {
'MATH': true,
'LITERAL': false,
'OBJECT': true
}
},
// ...
}
}
would disable the MATH
and OBJECT
mutations, while the LITERAL
mutations are still active.
In some cases, more fine-tuning is needed for which mutations should be excluded where. It is possible to disable mutations on block level by prepending the code with a comment containing the @excludeMutations
keyword.
When only the @excludeMutations
comment is provided, all mutations will be excluded for the block above which that comment is placed. It is also possible to provide an array of specific mutations that should be excluded, e.g.:
function foo() {
// ...
}
/**
* @excludeMutations ['MATH', 'OBJECT']
*/
function bar() {
// ...
}
would disable the MATH
and OBJECT
mutations on the bar
method and its contents, but not on foo
.
All javascript comment types are supported, i.e. one can use both // @excludeMutations
and /* @excludeMutations ["ARRAY"] */
. These comments can also be placed in the middle of a line of code, object, or function call, to allow for very specific configuration.
In lieu of a formal style guide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality. Lint and test your code using Grunt.
after
lifecycle method;files
configuration being merged instead of overwritten;ALL
log configuration option for Karma not being allowed;options.successThreshold
not working in File view;'use strict';
is now ignored by default.ignore
now also works for file portions larger than just the mutations;discardReplacements
as ignoreReplacement
, which will ignore mutation replacements that match the configuration option:discardDefaultIgnore
config option in case the user wants to override the default value of options.ignore
.When upgrading from v0.x to v1.0, one will notice that a lot has changed. Most important of these changes is the new way in which the grunt-mutation-test task needs to be configured.
Rather than supplying a map of output files to a list of files that should be mutated, one now needs to configure which files are needed to run the unit tests, which unit test files should be included, and which files should be mutated. This is done by supplying the code
, specs
, and mutate
options respectively. Take a look at the usage examples above for a few example configurations.
Copyright (c) 2015 Jimi van der Woning, Martin Koster. Licensed under the MIT license.
Original version by Marco Stahl