Allure Report shows incorrect calculations for failed tests because of test retries

[kirannandarapalli]

We have a retries(2) option that is configured in our framework to retry when we have a failed test. Recently, we upgraded the allure-cucumber gem from 2.14.3 to 2.23.0, and due to this we see the allure report showing all the retries in the report which leads to wrong calculation of failures

Also, the report shows that failure even though the test passed a second-time retry

Ex: Test 1 - Failed Retry Test 1 - Passed The report shows that both Test 1 passed and Retry 1 failed

[andrcuns]

I tried to reproduce the problem and don't really see an issue here. The report displays everything correctly with retry of a failed test. Consider following example from https://github.com/allure-examples/allure-cucumber-example:

$ bundle exec cucumber --retry 1 features/flaky.feature
Using the default profile...
Feature: flaky test example

  Scenario: Passes on the second try # features/flaky.feature:3
    Given I am a flaky test          # features/step_definitions/common_steps.rb:7
    When I run this test             # features/step_definitions/common_steps.rb:15
    Then it passes on the second try # features/step_definitions/common_steps.rb:31

      expected: 2
           got: 1

      (compared using ==)
       (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/common_steps.rb:32:in `"it passes on the second try"'
      features/flaky.feature:6:in `it passes on the second try'

  Scenario: Passes on the second try # features/flaky.feature:3
    Given I am a flaky test          # features/step_definitions/common_steps.rb:7
    When I run this test             # features/step_definitions/common_steps.rb:15
    Then it passes on the second try # features/step_definitions/common_steps.rb:31

1 scenario (1 flaky)
6 steps (1 failed, 5 passed)
0m0.017s

This will produce report with 1 passing example but with an indicator that status failed on retry:

The retry tab will contain first failed run:

This seems to be exactly the behavior expected from a retry.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days