search

pytest-dev / pytest-repeat

pytest plugin for repeating test execution
https://pypi.python.org/pypi/pytest-repeat/
Other
169 stars 27 forks source link

What problem does pytest-repeat solve? #47

Closed MartinThoma closed 4 years ago

MartinThoma commented 4 years ago

I've read / heard about this plugin a couple of times already, but I don't understand what it is good for. Why would you want to repeat a unit test several times? Which type of problem does it solve?

I would appreciate a small example :pray:

The-Compiler commented 4 years ago

I usually use it when I have some kind of flaky failure, i.e. I'm debugging an issue where I have a reproducer, but it only triggers the issue one out of 50 times. Then I can repeat that test until the failure is triggered (or repeat it until I'm confident enough that my fix actually works).

MartinThoma commented 4 years ago

Ok, makes sense.

I have seen this kind of behavior for parallel execution. Are there other groups of applications / use cases where this is happening? Do you add this then only when you have a good idea where the error is and remove this later as well, or is this then something that could land in the master branch?

The-Compiler commented 4 years ago

Happens a lot in asynchronous applications (in my case, a GUI application) as well. I imagine there are other ways things are sometimes non-deterministic in any larger project, though. Code related to randomness/probability/statistics comes to mind.

I personally only use --count on the command line while trying to reproduce something, and then usually try to find a better way to test things based on what I've found out while debugging. Others might use it differently. Also see a code search for the repeat decorator for some examples in the wild.

MartinThoma commented 4 years ago

Thank you very much!