The issue with SQL tests is that there is a test template class in the library (which contains SQL-related test setup code). This class is the parent class of the student's actual test class.
This parent class also matched the rules for test class discovery - its name contained the substring Test. As Andy does not expect that, it picks an arbitrary class matching that condition, meaning that on each execution either the actual test class or this dummy class containing no test methods will randomly be picked to be run, leading to the observed flakiness.
I fixed that by renaming the template class and adding a check to Andy to throw an exception if multiple matching classes are found, instead of simply picking one to make sure this does not happen in the future.
Added this test in an attempt to reproduce #114
The issue with SQL tests is that there is a test template class in the library (which contains SQL-related test setup code). This class is the parent class of the student's actual test class.
This parent class also matched the rules for test class discovery - its name contained the substring
Test
. As Andy does not expect that, it picks an arbitrary class matching that condition, meaning that on each execution either the actual test class or this dummy class containing no test methods will randomly be picked to be run, leading to the observed flakiness.I fixed that by renaming the template class and adding a check to Andy to throw an exception if multiple matching classes are found, instead of simply picking one to make sure this does not happen in the future.
Closes #114