search

proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.63k stars 152 forks source link

Persistence only written at program exit #439

Open vlovich opened 2 months ago

vlovich commented 2 months ago

I was racking my brain trying to figure out why each run yielded different results and I think I know what's happening. I run in a debugger, observe a failure (usually on the very first iteration), stop the program & rerun. Or I run it on the command-line, observe a bunch of failures & ctrl-c. However, it seems like proptest only saves the failure once on program exit which is annoying if the property test takes a while to run (yes I know I can configure the number of iterations manually to work around these issues).

It would be nice if it recorded the seed at the start of a run so that interruptions would be marked as such & that seed would get replayed on the next run until a complete successful run is done.

matthew-russo commented 2 months ago

Hey thanks for the report(s) -- I see you opened a few all related to ergonomics/UX around seed persistence/replay. i'll try to respond to each of them but if theres anything you think would be useful that's a minor breaking change, let us know as we're evaluating what things would go in a 2.0 release.

re: this issue specifically, would printing the seed at the start of the run be good enough or do you absolutely need it written to disk? adding a log would be trivial, adding a new type of persistence mode would be a bit more involved.

vlovich commented 2 months ago

Appreciate you looking at this. Printing it at the beginning is useful. Having it persisted at the beginning of a run would be a real nice to have so that the seed doesn't move as long as the test case is failing to finish.