allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
730 stars 238 forks source link

Duplicated test case statuses reported with pytest-rerunfailures, retries section is empty #735

Closed plume-aostrowski closed 1 year ago

plume-aostrowski commented 1 year ago

[//]: # ( . Note: for support questions, please use Stackoverflow or Gitter. . This repository's issues are reserved for feature requests and bug reports. . . In case of any problems with Allure Jenkins plugin please use the following repository . to create an issue: https://github.com/jenkinsci/allure-plugin/issues . . Make sure you have a clear name for your issue. The name should start with a capital . letter and no dot is required in the end of the sentence. An example of good issue names: . . - The report is broken in IE11 . - Add an ability to disable default plugins . - Support emoji in test descriptions )

I'm submitting a ...

What is the current behavior?

Tests that are re-run with pytset-rerunfailures are reported as multiple failures, while the "Retries" tab remains empty - as visible on the attached screenshot. Screenshot from 2023-03-09 14-17-34

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

The contents of the test script, in this case test_me_2.py:

def test_fail():
    assert False

Execute pytest --reruns=2 --alluredir=<allure_dir> test_me_2.py And then display the allure report.

The environment (venv) contents:

allure-pytest==2.13.0
allure-python-commons==2.13.0
attrs==22.2.0
exceptiongroup==1.1.0
iniconfig==2.0.0
packaging==23.0
pluggy==1.0.0
pytest==7.2.2
pytest-rerunfailures==11.1.2
six==1.16.0
tomli==2.0.1

What is the expected behavior?

Tests are reported once, in this case it is obviously a failure by design, the re-runs end up in the "Retries" section as on the screenshot below. This is the report I'm seeing when using allure-pytest-2.12.0: Screenshot from 2023-03-09 14-19-24

What is the motivation / use case for changing the behavior?

The plugin pytest-rerunfailures is a common pytest plugin to make an attempt at re-running flaky tests. Allure used to work perfectly with it, now it doesn't.

Please tell us about your environment:

Other information

[//]: # ( . e.g. detailed explanation, stacktraces, related issues, suggestions . how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc )

delatrie commented 1 year ago

Hi, @plume-aostrowski! Thanks for the report. The fix is on its way!

plume-aostrowski commented 1 year ago

Thanks for sorting it out so quickly!