Scenario retries are inconsistently tagged

[OnyinyeAladiume22]

We're seeing what appears to be a bug in the retry behavior of the parallel execution path. It seems, intermittently, identical scenarios that have been retried do not get logged as retries.

This is resulting in the total number of scenarios fluctuating and inconsistency in reporting. The below screenshot shows an example of a single launch that contains a scenario retried, and marked as such, appropriately. it also contains a launch that is identical, and was retried by the cucumber framework, but not registered as a retry by report portal.

The screenshot shows Scenario: Validate users can sign out was successfully retried and marked as such.

Alternatively, Scenario: Authenticated new user-Verify My list functionality shows as 2 unique entries even though it was the same scenario executed multiple times as a retry.

Note, this behavior has only been tested with scenarioBasedStatistics=true.

If there are any service level logs we can provide to help triage and resolve the issue, please let us know.

[OnyinyeAladiume22]

@AmsterGet @Bam6ycha any thoughts on this? Would you be the correct people to review?

[Bam6ycha]

@OnyinyeAladiume22, good day! It would be nice if you can provide reproducible repo for that issue. Because usually most of the time we spend reproducing issues. I hope you will understand.

[OnyinyeAladiume22]

@Bam6ycha we dont have an example repo readily available, especially for parallel execution conditions, to replicate this in but can look at creating one. in the meantime if there are any logs we can provide from our environment that would help debug please let us know. again, the repro conditions are

• running any amount of parallel threads
• with scenariobasedstatistics = true
• observing scenario retry behavior

while the issue is intermittent among scenarios, it is consistently happening in every launch that has retries occuring

[HardNorth]

Hi @OnyinyeAladiume22, to investigate your issue we need more information. Could you please click on pencil icon right next to problematic test items and send us screens? Or just copy any field you will see there?

[OnyinyeAladiume22]

With these 2 first images, the scenario correctly captures retries.

However, in the same launch, one instance of the previous scenario, isn't captured in retry: corresponding test results for image 3:

Next, 2 scenarios that did not get grouped as retries, but were expected to be:

their corresponding test item details - not grouped as retries but have the same unique test case id. That's it. @HardNorth @Bam6ycha

[OnyinyeAladiume22]

as an additional data point we are seeing that, at the feature level (with scenarioBasedStatistics=true, Features that are the same, but have been retried, are having duplicates created with unique UUIDs. Are these expected to be created with same UUIDs so they get bucketed as retries (which they are) instead of unique scenario executions (which they are not)?

[OnyinyeAladiume22]

@HardNorth @Bam6ycha

[OnyinyeAladiume22]

@HardNorth @Bam6ycha Do you need any more info to triage the bug we reported?

[rutika89]

Is there any update on this issue? We are facing the same problem.

[AmsterGet]

Hi guys! @OnyinyeAladiume22 , @rutika89 It's difficult to investigate and fix the issue without constantly reproducible sample. We still didn't catch such flaky behavior on our data, so it would be really helpful to have a repo with issue example. Looking forward for the details from you!