Set Number of tests by time (idea)

HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

https://hypothesis.works

Other

7.57k stars 587 forks source link

Set Number of tests by time (idea) #2379

Closed zkessin closed 4 years ago

zkessin commented 4 years ago

Instead of saying @settings(max_examples=500000) it might be nice if we could say something like run this property for some unit of time. This way if someone wants to run millions of tests over the course of a weekend they could just say "run this for 2 days" and instead of stopping after some number of examples it will stop after some amount of time.

What do people think?

Zac-HD commented 4 years ago

Running by time-limit tends to be a bit flaky due to performance variation, but you can use our new fuzzing support to run tests indefinitely (and ctrl-C when done). This also makes it coverage-guided, which is really helpful for such long runs!