Concurrent execution of JUnit 5 tests leading to incorrect coverage computations and improper TIA

[pjmartos]

When using skippy for test impact analysis together with JUnit 5 tests, we're observing unpredictable coverage, wrong test classes being marked for execution, and test classes being skipped when they shouldn't, when either of the following happen:

• Test classes are running in concurrent parallel mode, and/or
• There are test methods scattered across two or more @Nested classes, or across a @Nested class and the top-level class itself, or both

I believe this is due to the way coverage computations are being leveraged to determine the mapping of production classes vs test classes that visit them: https://github.com/skippy-io/skippy/blob/main/skippy-core/src/main/java/io/skippy/core/SkippyTestApi.java#L115,L131

Lines 115 and 131 assume that test classes run fully isolated and that the coverage computations observed since the last reset (i.e. in scope of a beforeAll callback) must undoubtedly and unambiguously correspond to production code exercised by the test class that has just finished running and for which we're running an afterAll callback.

[fmck3516]

Thanks for the report.

It would be great if you could provide minimal examples for both bullet points (but I will try to re-produce the issue myself).

Your observation is spot on: Skippy currently assumes that tests within a JVM execute serially. I have to wrap my head around how to solve that.