Possible to run instrumented Android tests under mutation?

[jazwiecki]

Apologies if this is the wrong place for this question! I have a set of instrumented Android tests which run on an emulated device. I can run them with gradle using gradlew connectedDebugAndroidTest, and I've set up gradle-pitest-plugin like so:

buildscript {
    repositories {
        google()
        jcenter()
        mavenLocal()
        mavenCentral()
    }
    configurations.maybeCreate('pitest')
    dependencies {
        classpath 'com.android.tools.build:gradle:3.5.2'
        classpath 'com.google.ar.sceneform:plugin:1.13.0'
        classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.2.2'
    }
}

apply plugin: 'com.android.application'
apply plugin: 'pl.droidsonroids.pitest'

android {
    compileSdkVersion 29
    buildToolsVersion "29.0.2"
    defaultConfig {
        applicationId "my.application.id"
        minSdkVersion 28
        targetSdkVersion 29
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
        debug {
            testCoverageEnabled = true
        }
    }
    useLibrary 'android.test.runner'
    useLibrary 'android.test.base'
    useLibrary 'android.test.mock'
    compileOptions {
        sourceCompatibility = 1.8
        targetCompatibility = 1.8
    }
}

dependencies {
    // Provides ARCore Session and related resources.
    implementation 'com.google.ar:core:1.13.0'
    // Provides ArFragment, and other Sceneform UX resources:
    implementation "com.google.ar.sceneform.ux:sceneform-ux:1.13.0"

    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'androidx.appcompat:appcompat:1.0.2'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    testImplementation 'junit:junit:4.12'
    testImplementation "org.mockito:mockito-core:+"

    // AndroidX Test dependencies
    // Core library
    androidTestImplementation 'androidx.test:core:1.0.0'

    // AndroidJUnitRunner and JUnit Rules
    androidTestImplementation 'androidx.test:runner:1.1.0'
    androidTestImplementation 'androidx.test:rules:1.1.0'

    // Assertions
    androidTestImplementation 'androidx.test.ext:junit:1.0.0'
    androidTestImplementation 'androidx.test.ext:truth:1.0.0'
    androidTestImplementation 'com.google.truth:truth:0.42'

    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test.espresso:espresso-contrib:3.1.0'
    androidTestImplementation 'androidx.test.espresso:espresso-intents:3.1.0'
    androidTestImplementation 'androidx.test.espresso:espresso-accessibility:3.1.0'
    androidTestImplementation 'androidx.test.espresso:espresso-web:3.1.0'
    androidTestImplementation 'androidx.test.espresso.idling:idling-concurrent:3.1.0'

    // The following Espresso dependency can be either "implementation"
    // or "androidTestImplementation", depending on whether you want the
    // dependency to appear on your APK's compile classpath or the test APK
    // classpath.
    androidTestImplementation 'androidx.test.espresso:espresso-idling-resource:3.1.0'
    androidTestImplementation 'androidx.test.uiautomator:uiautomator:2.2.0'
    androidTestImplementation "org.mockito:mockito-android:+"
    implementation 'org.pitest:pitest:1.4.5'
}

pitest {
    targetClasses = ['class.to.test.*']
    threads = 5
    outputFormats = ['HTML']
    verbose = true
}

When I run gradlew pitest or gradlew pitestDebug, the emulator doesn't start, and only my unit tests run. Specifying the instrumented test class in the pitest config or specifying a different test runner doesn't help. I'm new to Android Studio and configuring mutation testing with gradle, so I'm not sure if I'm missing something simple or this absolutely isn't possible.

The test class which isn't being run is decorated with @RunWith(AndroidJUnit4.class) (from androidx.test.ext.junit.runners.AndroidJUnit4).

I took a blind swing at specifying testPlugin = 'AndroidJUnitRunner' in the pitest config but that only gave me errors like this:

10:49:15 PM PIT >> INFO : MINION : 10:49:15 PM PIT >> SEVERE : Error calculating coverage. Process will exit.
org.pitest.util.PitError: Could not load requested test plugin androidx.test.runner.AndroidJUnitRunner

Please copy and paste the information and the complete stacktrace below when 
10:49:15 PM PIT >> INFO : MINION : reporting an issue
VM : Java HotSpot(TM) 64-Bit Server VM
Vendor : Oracle Corporation
Version : 25.92-b14
Uptime : 239
Input -> 
 1 : -Djava.awt.headless=true
 2 : -javaagent:/var/folders/0n/8fb29gnx38n5bbt5_r5yqg4m0000gn/T/15756041547020590626444224307220
10:49:15 PM PIT >> INFO : MINION : 91768261800559874.jar
 3 : -Dfile.encoding=UTF-8
 4 : -Duser.country=US
 5 : -Duser.language=en
 6 : -Duser.variant
BootClassPathSupported : true

...which wasn't really a surprise, because AndroidJUnitRunner isn't a pitest plugin, but I was hoping it might work. Any suggestions?

[koral--]

That seems doable, however it won't just work that way. This plugin is only a wrapper around pitest: https://pitest.org/ Pitest perform mutations and corresponding unit tests itself. So in order to have mutation instrumented test we need a tool which mutates the code, builds mutated apk/aab, performs tests and collects results.

[jazwiecki]

So that would be a plug-in for Pitest?

On Dec 9, 2019, at 11:59 AM, Karol Wrótniak notifications@github.com wrote:

 That seems doable, however it won't just work that way. This plugin is only a wrapper around pitest: https://pitest.org/ Pitest perform mutations and corresponding unit tests itself. So in order to have mutation instrumented test we need a tool which mutates the code, builds mutated apk/aab, performs tests and collects results.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[koral--]

Yeah, ideally it should be a plugin.

[jazwiecki]

OK, thank you very much!