Flaky tests are common occurrences in open-source projects, yielding inconsistent results—sometimes passing and sometimes failing—without code changes. NonDex is a tool for detecting and debugging wrong assumptions on under-determined Java APIs. I have resolved a flaky test issue using NonDex tool, specifically in the MethodSorterTestclass located at org.junit.internal.MethodSorterTest.testJvmMethodSorter
Root cause
The flakiness comes from the original codes which tends to sort the list of methods of a class, and use this test to test whether the declared methods of a class is gotten in a predictable order after going through his method sorter. However, the codes here firstly gets the first list of methods using java.lang.Class.getDeclaredMethods(), then gets the second one by making the class of methods going through the sorter, notice that the sorter is written based on a comparator that compare the 'methods' words by getting their hashcode. The fact is, java.lang.Class.getDeclaredMethods() method in Java does not guarantee a specific order for the returned methods. The order in which the methods are returned is not specified and may vary between different Java implementations or versions.
Fix
The logical of comparator is correct, the test is made flaky only because of java.lang.Class.getDeclaredMethods(), thus remove that function to get the methods' list, instead directly apply the comparator's logic in the test to make it work normally.
Setup
Java: openjdk version "11.0.20.1"
Maven: Apache Maven 3.6.3
How to test
Compile the module
mvn install -pl . -am -DskipTests
Run regular tests
mvn -pl . test -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSorter
Run tests with NonDex tool
mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSorter
Description
Flaky tests are common occurrences in open-source projects, yielding inconsistent results—sometimes passing and sometimes failing—without code changes. NonDex is a tool for detecting and debugging wrong assumptions on under-determined Java APIs. I have resolved a flaky test issue using NonDex tool, specifically in the
MethodSorterTestclass located atorg.junit.internal.MethodSorterTest.testJvmMethodSorterRoot cause
The flakiness comes from the original codes which tends to sort the list of methods of a class, and use this test to test whether the declared methods of a class is gotten in a predictable order after going through his method sorter. However, the codes here firstly gets the first list of methods using
java.lang.Class.getDeclaredMethods(), then gets the second one by making the class of methods going through the sorter, notice that the sorter is written based on a comparator that compare the 'methods' words by getting their hashcode. The fact is,java.lang.Class.getDeclaredMethods()method in Java does not guarantee a specific order for the returned methods. The order in which the methods are returned is not specified and may vary between different Java implementations or versions.Fix
The logical of comparator is correct, the test is made flaky only because of
java.lang.Class.getDeclaredMethods(), thus remove that function to get the methods' list, instead directly apply the comparator's logic in the test to make it work normally.Setup
Java: openjdk version "11.0.20.1" Maven: Apache Maven 3.6.3
How to test
mvn install -pl . -am -DskipTestsmvn -pl . test -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSortermvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSorterAfter fix, all tests pass