Build not failing when `mutationThreshold` is set

[magneticflux-]

Plugin version: 1.1.11 Java version: 1.8.0_111 Test suite: Unit 4.11 Gradle version: 3.4.1

Expected outcome: Gradle build fails after running pitest task. Actual outcome: Gradle build succeeds even with 3 mutations passing and mutationThreshold set to 0 or 1

[szpak]

I haven't used that feature in my projects, but I thought it is a value at which (or below) the build should be failed (something like minimal line coverage). Could you set it to higher value to check my theory?

If not please verify with --info that the value is passed to PIT and please paste the PIT summary report (with number of survived mutations, etc.)

[magneticflux-]

Here is the project that I am trying to use it in: magneticflux-/MinesweeperJ. I have tried 0, 1, and 10 when I have a total of 3 living mutations, and none of them fail even after a full clean and rebuild between each value change.

I believe that the value represents the maximum number of living mutations that still passes the build. So if you go over that value, your build fails.

Here is the report:

Executing external task 'pitest'...
:compileJava
:processResources NO-SOURCE
:classes
:compileTestJava
:processTestResources NO-SOURCE
:testClasses
5:51:47 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
5:51:47 PM PIT >> INFO : MINION : objc[70256]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will b
5:51:47 PM PIT >> INFO : MINION : e used. Which one is undefined.

5:51:48 PM PIT >> INFO : Sending 1 test classes to minion
5:51:48 PM PIT >> INFO : Sent tests to minion
5:51:48 PM PIT >> INFO : MINION : 5:51:48 PM PIT >> INFO : Checking environment

5:51:48 PM PIT >> INFO : MINION : 5:51:48 PM PIT >> INFO : Found  3 tests

5:51:48 PM PIT >> INFO : MINION : 5:51:48 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

5:51:48 PM PIT >> INFO : MINION : 5:51:48 PM PIT >> INFO : 3 tests received

:pitest
5:51:48 PM PIT >> INFO : Calculated coverage in 1 seconds.
5:51:49 PM PIT >> INFO : Created  1 mutation test units

\stderr  : objc[70443]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will bstderr  : e used. Which one is undefined.
5:51:50 PM PIT >> INFO : Completed in 2 seconds

/================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 1 seconds
> build mutation tests : < 1 second
> run mutation analysis : 1 seconds
--------------------------------------------------------------------------------
> Total  : 2 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 4 mutations Killed 3 (75%)
>> Ran 5 tests (1.25 tests per mutation)
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 0 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 2 Killed 2 (100%)
> KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 0 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 0 
--------------------------------------------------------------------------------

BUILD SUCCESSFUL

Total time: 4.947 secs

[magneticflux-]

It seems as though the maxSurviving parameter does not change the output either, testing 0 and 10.

[magneticflux-]

@szpak Is there a way to provide a raw arguments list to the PIT executable as a workaround?

[szpak]

Not directly.

However, I don't see a part of output with --info. It can be a link to Travis or copy/paste of lines where the plugin logs exact parameters passed to PIT (to verify the problem is there).

[magneticflux-]

I'm not seeing any mention of a --info flag. Do you mean verbose logging? Here is a Travis build with the verbose flag enabled: link.

[szpak]

@magneticflux- I've checked it locally and the situation is as I suggested in my first reply.

In your case there is:

Generated 4 mutations Killed 3 (75%) Ran 5 tests (1.25 tests per mutation)

When you set mutationThreshold = 80 your build will fail with:

Exception in thread "main" java.lang.RuntimeException: Mutation score of 75 is below threshold of 80
        at org.pitest.mutationtest.commandline.MutationCoverageReport.throwErrorIfScoreBelowMutationThreshold(MutationCoverageReport.java:68)
        at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:49)
:pitest FAILED

Does it solve your issue?

[szpak]

You may want to ask Henry (on the PIT mailing list) to enhance a description:

--mutationThreshold
Mutation score threshold at which to fail build.

on http://pitest.org/quickstart/commandline/

I also had some doubts reading it for the first time (what exactly "mutationScore" is). Although a comparison to coverageThreshold is helpful.

[magneticflux-]

@szpak Yes, I see now that the value is neither an absolute limit on surviving mutations, nor a percentage float as was my second guess. Thank you for your help! I'll send word to him with some suggestions to enhance the documentation.