search

str0zzapreti / pytest-retry

A simple plugin for retrying flaky tests in CI environments
MIT License
27 stars 6 forks source link

feat: --max-global-retries option #37

Closed deronnax closed 3 weeks ago

deronnax commented 4 months ago

Hello. Thank you for the plugin, amazing plugin. We have a usecase that is not covered by the current status: we would like the global number of attempted retries in a whole test run to be at max --max-global-retries. It allows to better control the degree of flakiness of the whole test suite. I came out with this naive implementation, please review, I am all open to any modification. Thank you

str0zzapreti commented 4 months ago

@deronnax Thanks, I'm glad you're finding it useful! Just to make sure I understand, the idea is to set a limit on the total number of tests retried within an entire suite, after which point no further tests would be retried, correct?

E.g.: I have a suite of 10 tests and a global max retry of 5 The first test passes The second test fails and is retried twice The third test fails and is retried twice The fourth test fails and although it is configured to retry twice, after the first retry the global limit is reached Now if test 6 fails, it will not be retried even if it is configured to do so because of the global limit

I must say, this is a use case I haven't encountered before, but I do think it's an interesting option to add. Thank you for the proposed contribution, I'll review your changes as soon as I can

deronnax commented 4 months ago

Yes, that's exactly this. Thank you.