szpak / gradle-pitest-plugin

Gradle plugin for PIT Mutation Testing
http://gradle-pitest-plugin.solidsoft.info/
212 stars 57 forks source link

Incremental analysis of same code producing varying history files #233

Closed jose-antony-by closed 3 years ago

jose-antony-by commented 3 years ago

I'm trying to get incremental analysis working in github actions. When I'm running locally, the incremental analysis works as expected. The input and output file remains the same when the test and source code remains the same between pitest runs.

When running in GitHub runner (ubuntu docker image) the history file before and after is different across multiple runs.

When looking at the base64 encoded content. most of the lines remain the same between the runs, but there are slight diffs in about 10% or so of the lines.

Gradle Configuration:

pitest {
    testPlugin = 'junit5'
    targetClasses = ['com.company.*']
    mainSourceSets = [sourceSets.main]
    testSourceSets = [sourceSets.test]
    threads = Runtime.runtime.availableProcessors()
    timestampedReports = false
    useClasspathFile = true   
    timeoutConstInMillis = 10000
    outputFormats = ['XML', 'HTML']
    mutationThreshold = 90  
    coverageThreshold = 95
    excludedMethods = ['equals','hashCode','toString']
    historyInputLocation = "${buildDir}/pitest/pitest.history"
    historyOutputLocation = "${buildDir}/pitest/pitest.history"
}

Just for reference, here is how I'm caching the history file in github actions.

  pitest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Cache pitest
        id: cache-pitest
        uses: actions/cache@v2
        with:
          path: service/build/pitest
          key: ${{ runner.os }}-pitest-cache

      - name: Set up JDK
        uses: actions/setup-java@v1
        with:
          java-version: 11

      - name: Execute pitest
        id: build
        run: ./gradlew pitest 

Is the cache machine-specific or is there something that could prevent this behavior?

jose-antony-by commented 3 years ago

sorry. posted in the wrong place. Created a new issue under pitest https://github.com/hcoles/pitest/issues/802