search

str0zzapreti / pytest-retry

A simple plugin for retrying flaky tests in CI environments
MIT License
29 stars 6 forks source link

allure report result broken, but ok #43

Open pashkatrick opened 4 months ago

pashkatrick commented 4 months ago

Firstable, thak you for your lib, it's great works with xdist instead of "rerunfailures" :)

to the problem: it's a matter of allure-reports. (for example) There is a case, it is not successful, then a restart is applied and then it is successful. The final result in report will indicate the status of the first launch - failure or broken (screenshot). Is it possible to make some kind of setting in the library or somehow independently on my side so that the result of the rerun is included in the report?

allure-report has a Retries section, actually rerunfailures works with it, there is no such problem there (but there is a problem with xdist) - I’m torn into two parts :)

I will be glad to any help and tips, thank you!

Screenshot 2024-05-24 at 09 55 24
pashkatrick commented 4 months ago

@str0zzapreti some feedback?

str0zzapreti commented 4 months ago

Not sure at first glance, I've never used allure-reports before. Do you know what allure-reports uses to detect retries? Perhaps it was done in conjunction with the team that works on rerunfailures, since they're tied to Pytest directly.

pashkatrick commented 4 months ago

thank for response!

maybe its related to flacky state there about allure works - i don't know exactly too, but

When you run the same test multiple times, Allure produces a separate test result file for each run. The test report then includes the status and other information from the latest file, with the other runs shown as retries.

str0zzapreti commented 4 months ago

It shouldn't anything to do with the flaky marker, but it wouldn't be difficult to test. Rerunfailures uses the same marker to configure retries.

A brief look at Allure's documentation seems to indicate that yellow corresponds to a 'broken' test status, where the value allure read wasn't in the known list. Pytest-retry adds a Retried status on top of the existing values as an intermediate step for Pytest's own report output, but the final reported status of the test will always be Passed or Failed, so I'm not sure why a test which passed after a retry would appear with the broken indicator in Allure.

At this point, I would need more details about your specific setup and/or steps to reproduce the issue in order to determine if a fix is possible.