Open Owlwasrowk opened 7 months ago
@Owlwasrowk thanks for the details. Karate doesn't yet support retries and this issue is a great example of what all edge cases come in the way. the best option here is for someone to go through the details (I think you have done some good analysis already) and contribute a PR
@ptrthomas Thank you for your prompt response. I understand that Karate does not currently support retries out-of-the-box, and the complexities that come with implementing such a feature.
I appreciate your suggestion to contribute a PR to address this issue. The missing tags may be related to the Suite
object not retaining information about the scenario's originating example table during retries.
If you have any starting points or insights that could guide the development of a solution, I would be grateful for your advice.
@Owlwasrowk there is a developer guide that may help: https://github.com/karatelabs/karate/wiki/Developer-Guide
but anyone with Java experience should be able to dive in. I think you are spot on with the equals
implementation of scenario. we are missing the concept of what I propose to call "examplesParentIndex", ideally I would rename "examplesIndex" to "examplesRowIndex" and use "examplesIndex" for what's missing
here is where example index is inited: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/ScenarioOutline.java#L109
and here is where examples are created when parsing a feature: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/FeatureParser.java#L282
and now if you add one more component to the isEquals
check, this should work
Summary
Two issues have been observed after retrying failed scenarios in a test suite with a Scenario Outline that includes multiple Examples tables:
Steps to Reproduce
mvn clean test
. There is used a random flakyness to illustrate the behaviour.Expected Behavior
After retrying failed scenarios and updating the report:
Actual Behavior
After retrying failed scenarios and updating the results:
Environment
Additional Information
Relevance of the Bug
The tags in our Karate test scenarios are not merely for organizational purposes; they serve a critical role in integrating our test automation framework with our test management system. Each tag corresponds to a unique identifier in Xray, which allows us to map automated test results directly to the relevant test cases in our management system. The loss of tags from the Examples tables after retrying scenarios disrupts this integration.
Solution Hints
The problem with the titles appears to be related to how the
updateResults
method in theSuite
class compares existing scenarios to determine if they are equal. The comparison logic, which relies only on the section index and the example index, can erroneously consider different scenarios as equal under certain conditions.Mismatched Titles
The
updateResults
method (here) uses theequals
method of theScenario
class to match scenarios. TheisEqualTo
method (here) checks if the section index and the example index are the same. This can lead to a situation where different scenarios from separate Example tables are incorrectly identified as the same because they share the same index positions within their respective tables.A potential solution would be to refine the comparison logic to include a more distinctive identifier that takes into account the unique context of each scenario, ensuring that scenarios from different Example tables are not conflated.
Missing Tags
The issue with the missing tags after retrying scenarios is still under investigation.
Attachments
karate-reports
folder containing the generated report after the retry, showcasing the reported bug.