Gradle plugin for PIT Mutation Testing

The plugin provides an ability to perform a mutation testing and calculate a mutation coverage of a Gradle-based projects with PIT.

Quick start

The simplest way

Add gradle-pitest-plugin to the plugins configuration in your build.gradle file:

plugins {
    id 'java' //or 'java-library' - depending on your needs
    id 'info.solidsoft.pitest' version '1.15.0'
}

Call Gradle with pitest task:

gradle pitest

After the measurements a report created by PIT will be placed in ${PROJECT_DIR}/build/reports/pitest directory.

Optionally make it depend on build:

build.dependsOn 'pitest'

Note that when making pitest depend on another task, it must be referred to by name. Otherwise, Gradle will resolve pitest to the configuration and not the task.

Generic approach

"The plugins way" has some limitations. As the primary repository for the plugin is the Central Repository (aka Maven Central) it is also possible to add the plugin to your project using "the generic way":

buildscript {
    repositories {
        mavenCentral()
        //Needed only for SNAPSHOT versions
        //maven { url 'https://oss.sonatype.org/content/repositories/snapshots/' }
    }
    dependencies {
        classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.15.0'
    }
}

Apply the plugin:

apply plugin: 'java' //or 'java-library' - depending on your needs
apply plugin: 'info.solidsoft.pitest'

Plugin configuration

The Pitest plugin does not need to be additionally configured if you use JUnit 4. Customization is done in the pitest block:

pitest {
    targetClasses = ['our.base.package.*']  //by default "${project.group}.*"
    pitestVersion = '1.15.0' //not needed when a default PIT version should be used
    threads = 4
    outputFormats = ['XML', 'HTML']
    timestampedReports = false
}

The configuration in Gradle is the real Groovy code which makes all assignments very intuitive. All values expected by PIT should be passed as a corresponding types. There is only one important difference. For the parameters where PIT expects a coma separated list of strings in a Gradle configuration a list of strings should be used (see outputFormats in the example above).

Check PIT documentation for a list of all available command line parameters. The expected parameter format in a plugin configuration can be taken from PitestPluginExtension.

To make life easier taskClasspath, mutableCodePaths, sourceDirs, reportDir, verbosity and pitestVersion are automatically set by the plugin. In addition sourceDirs, reportDir, verbosity and pitestVersion can be overridden by a user.

There are a few parameters specific for Gradle plugin:

• testSourceSets - defines test source sets which should be used by PIT (by default sourceSets.test, but allows to add integration tests located in a different source set)
• mainSourceSets - defines main source sets which should be used by PIT (by default sourceSets.main)
• mainProcessJvmArgs - JVM arguments to be used when launching the main PIT process; make a note that PIT itself launches another Java processes for mutation testing execution and usually jvmArgs should be used to for example increase maximum memory size (see #7);
• additionalMutableCodePaths - additional classes to mutate (useful for integration tests with production code in a different module - see #25)
• useClasspathFile - enables passing additional classpath as a file content (useful for Windows users with lots of classpath elements, disabled by default)
• fileExtensionsToFilter - provides ability to filter additional file extensions from PIT classpath (see #53)

For example:

pitest {
    ...
    testSourceSets = [sourceSets.test, sourceSets.integrationTest]
    mainSourceSets = [sourceSets.main, sourceSets.additionalMain]
    jvmArgs = ['-Xmx1024m']
    useClasspathFile = true     //useful with bigger projects on Windows
    fileExtensionsToFilter.addAll('xml', 'orbit')
}

Test system properties

PIT executes tests in a JVM independent of the JVM used by Gradle to execute tests. If your tests require some system properties, you have to pass them to PIT as the plugin won't do it for you:

test {
    systemProperty 'spring.test.constructor.autowire.mode', 'all'
}

pitest {
    jvmArgs = ['-Dspring.test.constructor.autowire.mode=all']
}

Eliminate warning in IDEA (Groovy-only)

As reported in #170 IntelliJ IDEA displays warnings about setting final fields (of lazy configuration) in build.gradle. It is not a real problem as Gradle internally intercepts those calls and use a setter instead. Nevertheless, people which prefer to have no (less) warnings at the cost of less readable code can use setters instead, e.g:

    testSourceSets.set([sourceSets.test, sourceSets.integrationTest])
    mainSourceSets.set([sourceSets.main, sourceSets.additionalMain])
    jvmArgs.set(['-Xmx1024m'])
    useClasspathFile.set(true)     //useful with bigger projects on Windows
    fileExtensionsToFilter.addAll('xml', 'orbit')

Multi-module projects support

gradle-pitest-plugin can be used in multi-module projects. The gradle-pitest-plugin dependency should be added to the buildscript configuration in the root project while the plugin has to be applied in all subprojects which should be processed with PIT. A sample snippet from build.gradle located for the root project:

//in root project configuration
plugins {
    id 'info.solidsoft.pitest' version '1.15.0' apply false
}

subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        threads = 4

        if (project.name in ['module-without-any-test']) {
            failWhenNoMutations = false
        }
    }
}

It is possible to aggregate pitest report for multi-module project using plugin info.solidsoft.pitest.aggregator and task pitestReportAggregate. Root project must be properly configured to use pitestReportAggregate :

//in root project configuration
plugins {
    id 'info.solidsoft.pitest' version '1.15.0' apply false
}

apply plugin: 'info.solidsoft.pitest.aggregator' // to 'pitestReportAggregate' appear

subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        // export mutations.xml and line coverage for aggregation
        outputFormats = ["XML"]
        exportLineCoverage = true
        timestampedReports = false
        ...
        reportAggregator {  //since 1.9.11 - extra results validation, if needed
            testStrengthThreshold.set(50)   //simpler Groovy syntax (testStrengthThreshold = 50) does not seem to be supported for nested properties
            mutationThreshold.set(40)
            maxSurviving.set(3)
        }
    }
}

After the pitest pitestReportAggregate tasks execution, the aggregated report will be placed in the ${PROJECT_DIR}/build/reports/pitest directory.

Integration tests in separate subproject

It is possible to mutate code located in different subproject. Gradle internally does not rely on output directory from other subproject, but builds JAR and uses classes from it. For PIT those are two different sets of class files, so to make it work it is required to define both mainSourceSets and additionalMutableCodePaths. For example:

configure(project(':itest')) {
    apply plugin: 'info.solidsoft.pitest'
    dependencies {
        implementation project(':shared')
    }

    configurations { mutableCodeBase { transitive false } }
    dependencies { mutableCodeBase project(':shared') }
    pitest {
        mainSourceSets = [project.sourceSets.main, project(':shared').sourceSets.main]
        additionalMutableCodePaths = [configurations.mutableCodeBase.singleFile]
    }
}

The above is the way recommended by the Gradle team, but in specific cases the simpler solution should also work:

configure(project(':itest')) {
    apply plugin: 'info.solidsoft.pitest'
    dependencies {
        implementation project(':shared')
    }

    pitest {
        mainSourceSets = [project.sourceSets.main, project(':shared').sourceSets.main]
        additionalMutableCodePaths = project(':shared').jar.outputs.files.getFiles()
    }
}

Minimal working multi-project build is available in functional tests suite.

PIT test-plugins support

Test plugins are used to support different test frameworks than JUnit4.

JUnit 5 plugin for PIT support (gradle-pitest-plugin 1.4.7+)

Starting with this release the configuration required to use PIT with JUnit 5 has been simplified to the following:

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.15.0'
}

pitest {
    //adds dependency to org.pitest:pitest-junit5-plugin and sets "testPlugin" to "junit5"
    junit5PluginVersion = '1.0.0'    //or 0.15 for PIT <1.9.0
    // ...
}

Please note. PIT 1.9.0 requires pitest-junit5-plugin 1.0.0+. JUnit Jupiter 5.8 (JUnit Platform 1.8) requires pitest-junit5-plugin 0.15+, while 5.7 (1.7) requires 0.14. Set right plugin version for JUnit 5 version used in your project to avoid runtime errors (such as `NoSuchMethodError: 'java.util.Optional org.junit.platform.commons.util.AnnotationUtils.findAnnotation(java.lang.Class, java.lang.Class, boolean)')).

The minimal working example for JUnit 5 is available in the functional tests suite.

For mixing JUnit 5 with other PIT plugins, you can read this section in my blog post.

Generic plugin support (also JUnit 5 in gradle-pitest-plugin <1.4.7)

To enable PIT plugins, it is enough to add it to the pitest configuration in the buildscript closure. For example:

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.15.0'
}

repositories {
    mavenCentral()
}

dependencies {
    pitest 'org.example.pit.plugins:pitest-custom-plugin:0.42'
}

The minimal working example is available in the functional tests suite.

Please note. In gradle-pitest-plugin <1.5.0 the pitest configuration had to be created in the buildscript scope for the root project. Please note. Starting with PIT 1.6.7 it is no longer needed to set testPlugin configuration parameter. It is also deprecated in the Gradle plugin.

Versions

Every gradle-pitest-plugin version by default uses a predefined PIT version. Usually this the latest released version of PIT available at the time of releasing a plugin version. It can be overridden by using pitestVersion parameter in a pitest configuration closure.

Please be aware that in some cases there could be some issues when using non default PIT versions.

If not stated otherwise, gradle-pitest-plugin 1.9.x by default uses PIT 1.9.x, 1.7.x uses PIT 1.7.x, etc. The minimal supported PIT version is 1.7.1.

Starting with version 1.7.0 gradle-pitest-plugin requires Gradle 6.4. The latest version with the Gradle 5.x (5.6+) support is 1.6.0. The current version was automatically smoke tested with Gradle 6.4, 6.9.1 and 7.4.2 under Java 11. Tests with Java 11+ are limited to the compatible versions of Gradle and PIT.

The experimental support for Java 17 can be tested with 1.7.0+.

Starting with the version 1.3.0 the produced binaries require Java 8 (as a JDK used for running a Gradle build). However, having Java 17 LTS released in September 2021, starting with gradle-pitest-plugin 1.9.0, support for JDK <11 is deprecated (see #299).

See the changelog file for more detailed list of changes in the plugin itself.

FAQ

How can I override plugin configuration from command line/system properties?

Gradle does not provide a built-in way to override plugin configuration via command line, but gradle-override-plugin can be used to do that.

After applied gradle-override-plugin in your project it is possible to do following:

./gradlew pitest -Doverride.pitest.reportDir=build/pitReport -Doverride.pitest.threads=8

Note. The mechanism should work fine for String and numeric properties, but there are limitations with support of Lists/Sets/Maps and Boolean values.

For more information see project web page.

How can I change PIT version from default to just released the newest one?

gradle-pitest-plugin by default uses a corresponding PIT version (with the same number). The plugin is released only if there are internal changes or there is a need to adjust to changes in newer PIT version. There is a dedicated mechanism to allow to use the latest PIT version (e.g, a bugfix release) or to downgrade PIT in case of detected issues. To override a default version it is enough to set pitestVersion property in the pitest configuration closure.

pitest {
    pitestVersion = '2.8.1-the.greatest.one'
}

In case of errors detected when the latest available version of the plugin is used with newer PIT version please raise an issue.

How to disable placing PIT reports in time-based subfolders?

Placing PIT reports directly in ${PROJECT_DIR}/build/reports/pitest can be enabled with timestampedReports configuration property:

pitest {
    timestampedReports = false
}

How can I debug a gradle-pitest-plugin execution or a PIT process execution itself in a Gradle build?

Occasionally, it may be useful to debug a gradle-pitest-plugin execution or a PIT execution itself (e.g. NPE in PIT) to provide sensible error report.

The gradle-pitest-plugin execution can be remotely debugged with adding -Dorg.gradle.debug=true to the command line.

However, as PIT is started as a separate process to debug its execution the following arguments need to be added to the plugin configuration:

pitest {
    mainProcessJvmArgs = ['-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005']
}

Can I use gradle-pitest-plugin with my Android application?

Short answer is: not directly. Due to some incompatibilities between "standard" Java applications and Android Java applications in Gradle the plugin does not support the later. Luckily, there is an Android fork of the plugin maintained by Karol Wrótniak which provides a modified version supporting Android applications (but on the other hand it doesn't work with standard Java applications).

I have JUnit 5 plugin and the execution fails after migration to 1.5.0+, why?

gradle-pitest plugin 1.5.0 finally relaxed the way how (where) the pitest configuration has been placed (#62) which also was generating deprecation warnings in Gradle 6+. This change is not backward compatible and as a result manual migration has to be made - see the release notes. This affects only project with external custom plugins.

Important. As the JUnit 5 plugin for PIT is definitely the most popular, starting with 1.4.7 there is a simplified way how it could be configured with junit5PluginVersion (which is definitely recommended). See my blog post to find out how to migrate (it also solves the compatibility issue with 1.5.0+).

Known issues

• too verbose output from PIT (also see verbosity option introduced with PIT 1.7.1)

Development

gradle-pitest-plugin cloned from the repository can be built using Gradle command:

./gradlew build

The easiest way to make a JAR with local changes visible in another project is to install it into the local Maven repository:

./gradlew install

There are also basic functional tests written using nebula-test which can be run with:

./gradlew funcTest

Support

gradle-pitest-plugin has been written by Marcin Zajączkowski with a help from contributors. The author can be contacted directly via email: mszpak ATT wp DOTT pl. There is also Marcin's blog available: Solid Soft - Working code is not enough.

The plugin surely has some bugs and missing features. They can be reported using an issue tracker. However, it is often a better idea to send a questions to the PIT mailing list first.

The plugin is licensed under the terms of the Apache License, Version 2.0.