hedgehogqa / fsharp-hedgehog

Release with confidence, state-of-the-art property testing for .NET.
https://hedgehogqa.github.io/fsharp-hedgehog/
Other
271 stars 31 forks source link

Run using the same seed until all tests pass #424

Open cmeeren opened 1 year ago

cmeeren commented 1 year ago

I came across a blog post on property testing that used jqwik. It seems to have some nice functionality that would be great to have in Hedgehog.

One useful feature is automatically rerunning falsified properties: If a property fails, jqwik by default uses the same seed for that property until it passes. It would be great to have this functionality in Hedgehog. The current behavior requires me to change check to recheck with the printed seed value for all failing properties to be sure that I am exposing the same failing behavior, and then change all the properties back after they pass. This is inconvenient.

I'm not sure how this can best be implemented, but I guess some temporary disk storage is needed for persistence between runs. Perhaps a temp folder with files named using a SHA256 hash of information that identifies the property (while being as robust as reasonably possible against changes to the source code), with the file contents being the seed value. If a property fails, the file is saved. If it passes, it is deleted (if it exists). Whenever a property is run, a seed is taken from the matching file, if it exists.