search

quarkiverse / quarkus-cucumber

Quarkus Cucumber extension
Apache License 2.0
14 stars 14 forks source link

Quarkus Continuous Testing re-runs all Features/Scenarios, although "only failed tests" is selected #138

Open erickloss opened 6 months ago

erickloss commented 6 months ago

Hello everyone,

I tried out the Quarkus Dev UI and Continuous Testing, which feels great so far. The Dynamic Test Cases that are created by the getTests() method seem to get the Dev UI in trouble. I will create and link a separate issue for that.

Also, when some of the scenarios fail and I choose "re-run only failed" via Dev UI or command line, all features are re-run. Since my project has a lot of features/scenarios, that slows down Continuous Testing massively in my case.

expected behavior: Only scenarios that failed are re-run when "only failed" is chosen

actual behavior: All scenarios are re-run if at least one scenario failed

----------- info on debugging:

I set a breakpoint in CucumberQuarkusTest::getTests and navigated the stack back to io.quarkus.deployment.dev.testing.JunitTestRunner.

These are all discovered Tests IDs:

image
  1. @TestFactory annotated method in Cucumber Mail Test class (the one with @CucumberOptions and so on.) extending CucumberQuarkusTest
  2. simple Jupiter Unit Tests
  3. idk ;)
  4. the class from 1. -> recognized via @QuarkusTest I guess
  5. more Unit Tests

When all tests are run, the internal state of JunitTestRunner has no additionalFilter set:

image

When only failed tests are re-run, there is a filter set:

image

The Quarkus Testing library seems to recognize the filter, maybe that can be used to filter dynamic tests in getTests()? I don't know how to access that state though.

I can contribute if you want and point me in the right direction.

Thank you Eric