Incremental analysis of same code producing varying history files

[jose-antony-by]

I'm trying to get incremental analysis working in github actions. When I'm running locally, the incremental analysis works as expected. The input and output file remains the same when the test and source code remains the same between pitest runs.

When running in GitHub runner (ubuntu docker image) the history file before and after is different across multiple runs.

When looking at the base64 encoded content. most of the lines remain the same between the runs, but there are slight diffs in about 10% or so of the lines.

Gradle Configuration:

pitest {
    testPlugin = 'junit5'
    targetClasses = ['com.company.*']
    mainSourceSets = [sourceSets.main]
    testSourceSets = [sourceSets.test]
    threads = Runtime.runtime.availableProcessors()
    timestampedReports = false
    useClasspathFile = true   
    timeoutConstInMillis = 10000
    outputFormats = ['XML', 'HTML']
    mutationThreshold = 90  
    coverageThreshold = 95
    excludedMethods = ['equals','hashCode','toString']
    historyInputLocation = "${buildDir}/pitest/pitest.history"
    historyOutputLocation = "${buildDir}/pitest/pitest.history"
}

Just for reference, here is how I'm caching the history file in github actions.

  pitest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Cache pitest
        id: cache-pitest
        uses: actions/cache@v2
        with:
          path: service/build/pitest
          key: ${{ runner.os }}-pitest-cache

      - name: Set up JDK
        uses: actions/setup-java@v1
        with:
          java-version: 11

      - name: Execute pitest
        id: build
        run: ./gradlew pitest 

Is the cache machine-specific or is there something that could prevent this behavior? If I run pitest consecutively in the same runner, the incremental scan also works.

[mikesmithson]

is there any traction on this issue? I've observed similar behavior.

[hcoles]

Sorry, completely missed this back in 2020.

The history file shouldn't be machine specific, but unfortunately its format is opaque and badly thought out so it is difficult to debug what's going on. It might be time to finally fix it.

What might be happening here is variation in the results due to timeouts and/or static initialisation code (see the Is It Random? section of the FAQ).

https://pitest.org/faq/

[mikesmithson]

Sorry, completely missed this back in 2020.

The history file shouldn't be machine specific, but unfortunately its format is opaque and badly thought out so it is difficult to debug what's going on. It might be time to finally fix it.

What might be happening here is variation in the results due to timeouts and/or static initialisation code (see the Is It Random? section of the FAQ).

https://pitest.org/faq/

@hcoles - OK, thanks. Is there a feature request that needs to be added for a future release for this? It's really hurting the productivity of our team.