str0zzapreti / pytest-retry

A simple plugin for retrying flaky tests in CI environments
MIT License
27 stars 6 forks source link

Add reporting functionality when using pytest xdist #29

Closed str0zzapreti closed 6 months ago

str0zzapreti commented 6 months ago

As noted in https://github.com/str0zzapreti/pytest-retry/issues/26, while pytest-retry is fully compatible with xdist from a test running standpoint, retry reports weren't displayed. Workers initialized from xdist don't call the terminal summary hook while the server doesn't run any tests, so nothing would output to the terminal.

Added a new server module and slightly refactored the RetryManager to store a report handler object rather than a simple StringIO stream.

The new OfflineReporter is the default when running sequentially without xdist. It encapsulates the existing report behavior although a slight change is that successful attempts for retried tests are now logged.

A ReportServer and ClientReporter handle building the report when xdist is invoked with n > 1. ClientReporters collect retry logs for each test as a group and then stream them to the ReportServer over a local socket. The ReportServer listens for incoming logs and assembles a complete retry report out of the incoming chunks. This ensures that retry logs for each test are grouped together in the final report for better readability. Ordering w/r/t overall test execution may obviously not be maintained, but that shouldn't matter.

Added a new test to validate that retry reports are output correctly when using xdist. Note that this test is not part of the normal pytest-retry CI suite for now and is mainly a tool for contributors to locally validate that their changes maintain compatibility with pytest-xdist. This is not a main priority.

Some other miscellaneous fixes: Added constants to the retry results for better clarity when reviewing the code and made a few minor name changes.