SERG-Delft / andy

Andy assesses student's test code. It's used in CSE1110, TU Delft.
MIT License
78 stars 22 forks source link

Fix issue with test discovery and add a system test for an SQL exercise #132

Closed martinmladenov closed 2 years ago

martinmladenov commented 2 years ago

Added this test in an attempt to reproduce #114

The issue with SQL tests is that there is a test template class in the library (which contains SQL-related test setup code). This class is the parent class of the student's actual test class.

This parent class also matched the rules for test class discovery - its name contained the substring Test. As Andy does not expect that, it picks an arbitrary class matching that condition, meaning that on each execution either the actual test class or this dummy class containing no test methods will randomly be picked to be run, leading to the observed flakiness.

I fixed that by renaming the template class and adding a check to Andy to throw an exception if multiple matching classes are found, instead of simply picking one to make sure this does not happen in the future.

Closes #114

martinmladenov commented 2 years ago

The test fails on the CI but passes locally, even in Docker... Odd.

martinmladenov commented 2 years ago

image

mauricioaniche commented 2 years ago

Great catch, not an easy bug to spot!