Fix flaky tests

[hofi1]

com.github.jk1.license.filter.LicenseBundleNormalizerSpec."The regular expression must be matched"

Problem

The test com.github.jk1.license.filter.LicenseBundleNormalizerSpec."The regular expression must be matched" asserts two JSON strings for equality. The corresponding data is stored in sets. But the order, in which the data is returned by the Sets, is not deterministic. The test expects the values to be in certain order, even tho two JSON String are considered as equal if the data is in a different order (regarding the specification of JSON). This leads to a flaky test.

This problem was found by the NonDex Engine.

https://github.com/jk1/Gradle-License-Report/blob/5e273f9753e2a3781dd58fa9db655c16d38484a3/src/main/groovy/com/github/jk1/license/Model.groovy#L39-L48

Solution

The best solution would be to use JsonAsserts. But since it is unlikely that the owern of the repo would accept a new dependency, it was fixed by making sure the data is returned in a deterministic order. By changing the HashSets to TreeSets (to guarantee the order of the elements in the JSON string) in the ModulData class and adding the @Sortable Annotation (needed by the TreeSet to order the elements in the set) to the PomData class, determinism is established.

https://github.com/jk1/Gradle-License-Report/blob/140ee4909f69d52df2ddc451a7df36b9a1718220/src/main/groovy/com/github/jk1/license/Model.groovy#L39-L48

https://github.com/jk1/Gradle-License-Report/blob/140ee4909f69d52df2ddc451a7df36b9a1718220/src/main/groovy/com/github/jk1/license/Model.groovy#L57-L64

com.github.jk1.license.ProjectBuilderSpec."it creates modules with license-files"

Problem

The test com.github.jk1.license.ProjectBuilderSpec."it creates modules with license-files asserts that two lists contain the same elements. These elements are stored in a List. But the order, in which the elements in the data structure are returned by the set, is not deterministic. The test expects the values to be in certain order. This leads to a flaky test.

This problem was found by the NonDex Engine.

https://github.com/jk1/Gradle-License-Report/blob/5e273f9753e2a3781dd58fa9db655c16d38484a3/src/test/groovy/com/github/jk1/license/ProjectBuilderSpec.groovy#L119-L121

Solution

By sorting the elements in the lists, a deterministic behavior can be guaranteed.

https://github.com/jk1/Gradle-License-Report/blob/140ee4909f69d52df2ddc451a7df36b9a1718220/src/test/groovy/com/github/jk1/license/ProjectBuilderSpec.groovy#L211-L213

Reproduce

To reproduce follow the steps:

1. Run ./gradlew build -x test
2. Add the following text to the top of the build.gradle file in $PROJ_DIR.

   buildscript {
   repositories {
     maven {
       url = uri('https://plugins.gradle.org/m2/')
     }
   }
   dependencies {
     classpath('edu.illinois:plugin:2.1.1')
   }
   }

3. Add the following line to the end of the build.gradle file in $PROJ_DIR.

   apply plugin: 'edu.illinois.nondex'

4. Run

   ./gradlew --info nondexTest --tests=_full.qulified.identification.of.test_ --nondexRuns=50

[jk1]

@hofi1 thank you for your contribution. I suppose that the tests are not actually flaky. The reason being, a HashMap ordering is not guaranteed, but is still stable from run to run. Yet it's definitely not something to rely upon in the long run.