Are you looking for ways to improve the quality and test coverage of your unit tests? Then Testura and mutation testing may be something for you.
Testura is a mutation testing tool/visual studio extension for C# that verifies the quality of your unit tests by injecting different mutations in your production code and then checks whether your unit tests catch them. If your unit tests:
Example of a mutation
The first thing you need to do is to configure the extension at Extensions > Testura.Mutation > Config..
.
You can run mutations in three different ways:
Extensions > Testura.Mutation > Mutate solution..
to mutate the whole solution (except for ignored/test projects).Mutate files...
Mutate lines
It may take a while to create your mutations depending on number of files/lines.
When they are done you see how the mutation look by clicking the small blue button the the right or double click and hover over the statemen (if you have highlight on).
Click the play icon in the mutation explorer to run your mutations. The mutations will either:
Please check the output window.
First you have to create a json run config, for example:
{
"SolutionPath": "path\\to\\your\\solution.sln",
"TestProjects": [
"*.Tests"
],
"BuildConfiguration": "Debug",
"TestRunner": "dotnet",
"NumberOfTestRunInstances": 5
}
Then you run the exe from for example cmd by writing:
path\to\Tetura.Mutation.Console.exe local --configPath "path/To/Json/Config" --outputPath "path/to/output/directory"
It will then:
After the unit test execution, you will get a couple of different result files (we plan to develop a filtering function in the future):
The result files contain your mutation score (number of killed mutants divided by the total number of mutants) as well as details about all survived mutants.
Path to your C# solution that we should mutate/run tests from.
A list of all test projects that we should run (we won't mutate test projects). For example:
"TestProjects": [
"MyTestProject"
"*.Tests"
],
If we should mutate "Debug" or "Release" (it's important to build your project before trying to run Testura)
Testura offers three different flags for the test runner:
In most cases it's best to run "dotnet".
We run all unit tests for all mutants and this property tells us how many sessions we should run in parallel.
List of projects that we shouldn't mutate or run tests from. For example:
"IgnoredProjects": [
"Name.Of.The.Project"
],
Filter is a way for you to decide which files or lines that Testura will mutate. You can both allow and deny resources, but it's important to know that when you add a filter, all files will be ignored by default and you have to allow them.
For example:
"Filter": {
"FilterItems": [
{
"Effect": "Allow",
"Resource": "*/src/some/files.cs",
"Lines": [
"59,10"
]
},
{
"Effect": "Allow",
"Resource": "*SuperFile*",
},
{
"Effect": "Deny",
"Resource": "*mock*"
},
{
"Effect": "Deny",
"Resource": "*",
"CodeConstrain": "*Log.*"
}
]
},
In this example, Testura will:
Log.
.We use filter a lot to run mutation operators on specific subsets for example new pull requests.
If you use dotnet as the test runner and Testura can't find dotnet.exe automatically it is possible to set it manually with this property.
A list of all the mutators that you want to run (look further below for tags and the default ones if you don't specifiy any mutators).
"Mutators": [ "Increment" ]
You can map test project(s) to mutation projects like this:
"ProjectMappings": [{
"ProjectName": "Testura.Code",
"TestProjectNames": ["Testura.Code.Tests"]
}
],
This means that if we create a mutation in the Testura.Code
project we will only verify it by executing tests in Testura.Code.Tests
.
Possible git values:
"Git": {
"GenerateFilterFromDiffWithMaster": false
},
GenerateFilterFromDiffWithMaster
: If this is true we will look at the changes between the branch and master and generate filter items for the changes. For example if you have changed line 51 in the file "Test.cs" we will create a filter item like { Effect: "Allow", "Resource": "*Test.cs", "Lines": [ "51" ]
A list of specific run loggers. For example:
"MutationRunLoggers": [
"Azure"
]
Current run loggers:
2019-05-16 14:50:29,251: Testura.Mutation.Core.Loggers.AzureMutationRunLogger: ##vso[task.setprogress value=67;]Mutation execution progress
Here is a short list of all currently available mutation operators:
Name | Mutations | Tag | Default |
---|---|---|---|
Conditional boundary | >= , >, <=, <, etc | ConditionalBoundary | true |
Math | +, -, *, etc | Math | true |
Increment | ++, -- | Increment | true |
Negate conditional | >, <, !=, ==, etc | NegateCondtional | true |
Negate type | i is bool, !(i is bool) | NegateTypeCompability | true |
Return value | return -1 instead of 0, return null instead of object, throw exception instead of returning null, etc | ReturnValue | true |
Void Method | Remove all calls to void methods | MethodCall | false |
This project is licensed under the MIT License. See the LICENSE.md file for details.
This work has been financially supported by the ITEA3 initiative TESTOMATProject through Vinnova - Sweden's innovation agency.