search

google / graphicsfuzz

A testing framework for automatically finding and simplifying bugs in graphics shader compilers.
Apache License 2.0
561 stars 117 forks source link

Fixed flaky test #1203

Open wang3820 opened 9 months ago

wang3820 commented 9 months ago

Flakiness

Nondex was used to check and locate the flakiness in the test. The test can be reproduced using the following command:

mvn install -pl generator -am -DskipTests
mvn -pl generator test -Dtest=com.graphicsfuzz.generator.util.TransformationProbabilitiesTest#testToString
mvn -pl generator edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.graphicsfuzz.generator.util.TransformationProbabilitiesTest#testToString

Issue

In the toString method, getDeclaredFields() was called to iterate through all the fields in this class. However, getDeclaredFields() does not guarantee the order of elements returned, according to the Java Documentation: The elements in the returned array are not sorted and are not in any particular order.

Fix

Since the output array of getDeclaredFields() is not in any particular order, the fix sorts the array of fields returned by getDeclaredFields() by field name and then convert the array to string with the original format, except that fields are sorted by name. In the testcase, the expected string is also sorted by field names. This way this testcase constantly passes, resolving any flakiness.

google-cla[bot] commented 9 months ago

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.