1.6.3 - Incorrect Scenario Outlines behavior

[unickq]

Hey there,

Cloned CucumberJvmExample repository and changed a bit ran following scenario:

@en
Feature: Test allure English

  Scenario: Test multiply
    Given first digit 5
    And second digit 5
    When I multiply it
    Then sum is 10

  Scenario Outline: Two digits multiply
    Given first digit <first>
    And second digit <second>
    When I multiply it
    Then sum is <sum>

    Examples: 
      | first  | second | sum   |
      | 6      | 6      | 11    |
      | 5      | 10     | 15    |

Expected: 3 scenarios in allure report: 1 passed Test multiply, 1 passed and 1 failed Two digits multiply Actual: Only last step of scenario outline scenario is published

mvn clean test

3 Scenarios (1 failed, 2 passed) 12 Steps (1 failed, 11 passed)

Allure 2.3.4 shows: 2 passed scenarios: 1) Test multiply 2) Two digits multiply | 5 | 10 | 15 |

[clicman]

Hi! Actually, it is expected behaviour, because scenario outline is the same test case with different dataset. But making it flaky is unexpected behavior. And somewhere I saw an issue on it.

[unickq]

@clicman So is it impossible to publish scenario outline data as a separate test case?

I mean how we can see failed dataset as well?

[clicman]

By design, it will be marked as flaky and you'll see which one failed in test history (retries)

[unickq]

Ok, thanks.

Just wondered why the last failed data set item fails the whole test case and last successful marks it passed even if other are different :)

    Examples: 
      | first  | second | sum   |
      | 6      | 6      | 12    |
      | 5      | 10     | 11    |

We can close this issue if that designed that way

[clicman]

Sometimes hard to integrate cucumber logic into allure. This is one of these difficulties )

[mikesalvia]

@clicman Maybe this can get re-opened, the Allure Cucumber Ruby adaptor is indeed able to take the parameters of the Scenario Outline and appends them to the Scenario Outline title.

[mikesalvia]

It looks like it should be being done here... However I am seeing them all grouped together.