Open pshevche opened 1 year ago
I also prepared a PR that should fix the issue, as it seemed pretty straight-forward. But let me know if that's not how you see it :).
I was also wondering, why did you decide to report such failures during the execution phase instead of making it a hard failure during the discovery phase?
Hi @pshevche, thank you for the report and especially for the PR. As far as I remember I did that weird workaround, because if it failed, the exception was somehow lost up in the Junit5 system during the discovery. As the result, no tests were "discovered" and the test run was "passing". I should investigate if it is still the issue.
@giurim , I see, it makes sense. Another benefit of this approach is that you could still execute as many test suites as possible and get at least some feedback instead of failing early, which might be a good thing.
If you decide to stick with the current strategy, then we would need to handle these custom descriptors one way or another, as it is documented as an engine requirement here: https://junit.org/junit5/docs/current/user-guide/#test-engines-requirements-mandatory (see the second point).
Describe the bug Currently, the engine will create a custom
TestDescriptor
if the test suite initialization fails (ref). This descriptor will have aUniqueId
of form[engine:scalatest]/[failed:<SUITE_NAME>]
. However, the engine does not support the suite discovery based on this ID. This limits the ways to interact with the engine, since the consumer can use the discovery result to further limit the tests to be executed and then pass a smaller set ofUniqueIdDescriptor
s toexecute
method.To Reproduce This branch has a test that shows the empty discovery result if the
UniqueId
of this syntheticTestDescriptor
is used as input.Expected behavior The engine detects that its synthetic descriptor is used and returns a valid discovery result.