Handling Inherited JUnit Dependencies in iDFlakies

[xyliu-cs]

Hi, I have noticed (also confirmed in the ICST’19 paper of iDFlakies) that iDFlakies checks whether the target module has the JUnit dependency declaration in its pom.xml file and skips the tests if not found.

However, I observed that some modules might have their tests written in JUnit but lacked an explicit declaration of the JUnit dependency in the pom.xml file. This happens because the module can inherit the JUnit dependency from its parent (in my case, it is the jetty-util module from jetty.project). iDFlakies seems to overlook this inheritance mechanism. When I manually added the JUnit dependency declaration in the jetty-util module's pom.xml, iDFlakies became able to identify and execute the tests, but it would simply skip this module without such modification. Unsurprisingly, it has discovered some new flaky tests within that module.

I have attached the steps to reproduce my experiment as commands.txt for your reference. Although I only explored this specific module, I believe it is a common issue for iDFlakies. It may be beneficial for iDFlakies to consider cases where JUnit dependencies are inherited from parent modules to avoid skipping valid tests. (maybe we can improve the dependency-checking part?) Is this an area for improvement?

commands.txt

[august782]

Thanks for reporting this. One thing I noticed about this jetty.project project is that the jetty-util module technically does not have a junit dependency defined, even when considering inheritance. The parent module defines a junit dependency under dependencyManagement, but not under dependencies, which means the child jetty-util module does not inherit any junit dependency under its own dependencies (if you run mvn dependency:tree for jetty-util, you'll see it only has junit-dep, not junit).

However, it does seem like one could run tests even without defining or inheriting any junit dependency. We may need to make some adjustments to force iDFlakies to use a JUnit 4 runner by default if not defined, but we also need to make sure we don't force it when the project has TestNG tests, which we currently do not support.

[xyliu-cs]

Thank you for your response. Upon further examination of the project, I have confirmed your conclusion. It appears that the jetty-util module does not inherit the junit dependency from its parent. Instead, the jetty-util module is getting the JUnit classes via the jetty-test-helper dependency, which, in turn, depends on junit-dep(discontinued in favor ofjunit). I apologize for my misinterpretation earlier. However, I'm glad it may still provide some useful information for you.