str0zzapreti
commented
1 year ago Hello, thanks for your interest in pytest-retry! Reporting issues were actually some of the main reasons I created this plugin, so I'm glad it seems to be working for you.
For context: flaky monkeypatches Pytest's internal testrunner loop, which kind of stretches the definition of "plugin" and makes it difficult to customize the behavior beyond the settings provided by the developers. You may have already seen it, but there's also pytest-rerunfailures which was built by some of the Pytest team and has a lot of community support. FWIW, I also had issues with the way it handles reporting and such, but I wanted to mention it just in case.
pytest-retry works a bit differently to the flaky and pytest-rerunfailures plugins because it hooks into Pytest's reporting functionality rather than rewriting the testrunner. If a test fails on the first attempt but then passes before the number of retries is exhausted, the original failed report data is overwritten with the results from the passing attempt. That way, generated reports (e.g. JUnit XML) should always show the expected results.
If you need information on which tests were retried and how many times, the plugin logs that data separately and makes it available both in the Pytest console report and from the stash stored on each test item. You can always write your own conftest hook to access the data and use it however you want.
Anyway, I hope that answers your question! I mainly built this plugin for myself, but that's the beauty of open source, really. If you encounter any problems, please feel free to open a new Github issue or PR and I'd be happy to take a look when I have time.
andreabisello
Hi, thanks for the plugin. what are the differences with flaky, a storical plugin to handle flaky test? to be honest i reached this plugin searching for an altenative to flaky for a better handling of the junit result. in fact, using flaky, the junit result contains the same raw both for successufly test and for retried test. In this way, i'm not able to understand what is working and what was retried. pytest-retry instead seems avoiding to put the retry xml temptative in the junit result, offering a clean situation. thanks