'allure-pytest-bdd' report marks feature as 'Fail' even when all steps pass when test is using 'Scenario Outline'

[ricwal-richa]

[//]: # ( . Note: for support questions, please use Stackoverflow or Gitter. . This repository's issues are reserved for feature requests and bug reports. . . In case of any problems with Allure Jenkins plugin please use the following repository . to create an issue: https://github.com/jenkinsci/allure-plugin/issues . . Make sure you have a clear name for your issue. The name should start with a capital . letter and no dot is required in the end of the sentence. An example of good issue names: . . - The report is broken in IE11 . - Add an ability to disable default plugins . - Support emoji in test descriptions )

I'm submitting a ...

• [X] bug report
• [ ] feature request
• [ ] support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

When generating an allure report for pytest-bdd test; the report shows test feature marked as Fail although all steps in the test are marked as Pass. The test is using Scenario Outline and running for 12 data sets.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

1. Create a test in pytest-bdd feature file. sample from my test -

   @regression
   Feature: Some description
   As blah
   
   Scenario Outline: value comparison
   Given the <feed_file> exists
   When the <attribute_name> with the <attribute_value> for the <customer_id> is provided in the <feed_file>
   Then verify that CDP mapping for <attribute_name> with <attribute_value> matches for the <customer_id>
   
   Examples:
     | feed_file | attribute_name  | attribute_value | customer_id |
     | Customer.csv  | FirstName  | Taylor  | 8470939 |
     | Customer.csv  | LastName  | Perry | 8470936 |
     | CustomerEmail.csv  | Email  | justsomeemail@gmail.com | 8470939 |

2. From terminal, run pytest --activedir=<absolute path to report folder>
3. Verify that multiple JSON files are generated under report folder when execution completes.
4. Open any one JSON file.
5. The report JSON shows feature status as 'Fail' even when all test step status shown as 'Pass'

   What is the expected behavior?

   Feature status should be marked as 'Pass' when all steps inside the test are 'Pass'.

What is the motivation / use case for changing the behavior?

Bug Fix

Please tell us about your environment:

• allure-pytest-bdd version: 2.8.6b0
• pytest version: 5.3.2
• pytest-bdd version: 3.2.0

Other information

. When same test is executed and report is generated using pytest-html, all tests show as Pass.

. Allure report screenshot

[fundakol]

I have reproduced the issue and found out the cause of it by looking into pytest_runtest_makereport method. It seems reduce function does not work as expected.

status = reduce(lambda final_status, current_status: final_status or getattr(report, current_status, None),
                        ["failed", "passed", "skipped"])

I have changed that piece of code with method from allure-pytest:

def get_pytest_report_status(pytest_report):
    pytest_statuses = ('failed', 'passed', 'skipped')
    statuses = (Status.FAILED, Status.PASSED, Status.SKIPPED)

    for pytest_status, status in zip(pytest_statuses, statuses):
        if getattr(pytest_report, pytest_status):
            return status

And now it works fine.

[ricwal-richa]

thanks @fundakol the fix works for me. Awaiting branch merge.