I tried out the Quarkus Dev UI and Continuous Testing, which feels great so far. The Dynamic Test Cases that are created by the getTests() method seem to get the Dev UI in trouble. I will create and link a separate issue for that.
Also, when some of the scenarios fail and I choose "re-run only failed" via Dev UI or command line, all features are re-run.
Since my project has a lot of features/scenarios, that slows down Continuous Testing massively in my case.
expected behavior:
Only scenarios that failed are re-run when "only failed" is chosen
actual behavior:
All scenarios are re-run if at least one scenario failed
----------- info on debugging:
I set a breakpoint in CucumberQuarkusTest::getTests and navigated the stack back to io.quarkus.deployment.dev.testing.JunitTestRunner.
These are all discovered Tests IDs:
@TestFactory annotated method in Cucumber Mail Test class (the one with @CucumberOptions and so on.) extending CucumberQuarkusTest
simple Jupiter Unit Tests
idk ;)
the class from 1. -> recognized via @QuarkusTest I guess
more Unit Tests
When all tests are run, the internal state of JunitTestRunner has no additionalFilter set:
When only failed tests are re-run, there is a filter set:
The Quarkus Testing library seems to recognize the filter, maybe that can be used to filter dynamic tests in getTests()? I don't know how to access that state though.
I can contribute if you want and point me in the right direction.
Hello everyone,
I tried out the Quarkus Dev UI and Continuous Testing, which feels great so far. The Dynamic Test Cases that are created by the
getTests()
method seem to get the Dev UI in trouble. I will create and link a separate issue for that.Also, when some of the scenarios fail and I choose "re-run only failed" via Dev UI or command line, all features are re-run. Since my project has a lot of features/scenarios, that slows down Continuous Testing massively in my case.
expected behavior: Only scenarios that failed are re-run when "only failed" is chosen
actual behavior: All scenarios are re-run if at least one scenario failed
----------- info on debugging:
I set a breakpoint in
CucumberQuarkusTest::getTests
and navigated the stack back toio.quarkus.deployment.dev.testing.JunitTestRunner
.These are all discovered Tests IDs:
@TestFactory
annotated method in Cucumber Mail Test class (the one with@CucumberOptions
and so on.) extending CucumberQuarkusTest@QuarkusTest
I guessWhen all tests are run, the internal state of JunitTestRunner has no
additionalFilter
set:When only failed tests are re-run, there is a filter set:
The Quarkus Testing library seems to recognize the filter, maybe that can be used to filter dynamic tests in
getTests()
? I don't know how to access that state though.I can contribute if you want and point me in the right direction.
Thank you Eric