Open class101 opened 1 year ago
On behalve of everyone who's contributed to this I'd like thank you for your praise. Personally I'm happy to see that you've been able to achieve such a significant performance improvement. I'm glad you've found it useful.
As for your issue.
Unfortunately the way Surefire implements rerunFailingTestsCount
is by starting a new execution of Cucumber, just for those failed tests. Currently Cucumber can't tell the difference and will happily overwrite any files it created in the first run, with the files from the second run.
Once JUnit 5 provides the ability for a test engine to store information in a session that spans both executions (i.e. https://github.com/junit-team/junit5/issues/2816) it should in theory be possible to write the report only once with the reruns included.
As a comparison, we're also using the excellent tool
maven-cucumber-reporting
by damianszczepanik and it seems to support the failsafe property just fine as you can see below :)
So this is quite puzzling. From what I understand, the plugin reads the json files produced by Cucumber. Those should have been overwritten. Perhaps there is a more subtle bug going on? From memory, the plugin merges multiple report files. Could it be possible that you are looking at json files produces by previous tests or different tests?
And a few notes for the future:
Surefire currently closes the launcher and launcher session for the initial attempt and every rerun. This means that even if JUnit 5 were to support sessions for test engines, Cucumber would not be aware of them. As Maven plugins tend to release and develop at a glacial speed, I would not expect it to be possible to report on reran scenarios 1-2 years until after JUnit implements support for it.
The core of Cucumber JVM currently does not support retrying tests. There is no good default strategy for retrying or rerunning tests, nor is there sensible way to report on those retried test. I'm loathe to deal with the fractal of complexity that comes with implementing one or more specific implementation of a rerun strategy
So while I'm happy to help make it possible to run the same scenario multiple times, I'd rather support this at a programmatic level only.
First, my deepest respects for the work you have done with cucumber. It is a simply fascinating tool.
I love it so much that when I arrived on the project on which I am today, the use that my team made of it challenged me.
They executed their
1450 scenarios
in6 hours
, after studying all your documentations and innovations such as@ScenarioScope
, we achieved a runtime of17min
, for the same 1450 tests!Well ok I admit that they also used too much
Thread.sleep
, which I transformed intoSelenium's Fluent Wait
;)It's one of the accomplishments I'm most proud of in my career and I wanted to share it with you ladies and gentlemen, keep up the excellent work :)
👓 What did you see?
When I use the following Maven property
failsafe.rerunFailingTestsCount
, property automatically recognized by themaven-failsafe-plugin
, I get the following actual incomplete report :✅ What did you expect to see?
When I remove the property, the report is as expected, 8 errors in total :
As a comparison, we're also using the excellent tool
maven-cucumber-reporting
by @damianszczepanik and it seems to support the failsafe property just fine as you can see below :)📦 Which tool/library version are you using?
I observe the problem on the last 2 versions, i.e.
7.11.0-SNAPSHOT
and7.10.1
, we are usingjDK 11,
Maven 3.8.7
and below the other libraries🔬 How could we reproduce it?
Steps to reproduce the behavior:
7.11.0-SNAPSHOT
📚 Any additional context?
We have tests that sometimes pass after the 2nd or 3rd attempt, that's why we would like to keep the support of the failsafe option, it avoids all the problems of false-positive test results that we had before, our cycle of execution is now 100% stable thanks to
rerunFailingTestsCount
We also think your work on cucumber's responsive HTML reporting interface is fantastic, great job.
Lastly, I don't think you have any problems reproducing the problem, but if I can ever help by trying to find the cause of the problem in the sources, I will be happy to help you digging into the source code, I have all set here to debug anywhere needed or by participating in a merge request, I am quite comfortable with the use of git.
I have taken care to send you as much information as possible so that the resolution is the most obvious. Maybe a little too much haha.
Cheers from France and happy new year 2023 everyone!
-- Arnaud Dovi HN Services / Crédit-Logement