stedolan / crowbar

Property fuzzing for OCaml
MIT License
183 stars 28 forks source link

With seed as cli parameter #60

Closed raphael-proust closed 3 years ago

raphael-proust commented 4 years ago

This PR adds a CLI parameter (-s/--seed) to make the quickcheck mode fully deterministic.

Note that the current state is mostly deterministic: two consecutive runs derive the same original seed from the global Random PRNG. However, linked libraries can change the PRNG state:

This PR is not yet tested. I wanted to get a quick mood check before putting significant effort in: Is this change or one like it likely to be merged? Is there interest for such a feature? Any comments on the user interface?

stedolan commented 4 years ago

Sounds good to me!

raphael-proust commented 4 years ago

:+1: I'll test this, make necessary corrections and update this PR.

raphael-proust commented 4 years ago

Good news: the seed flag works.


I also added a --repeat flag to set the number of repetitions in quickcheck mode. It also works.

It's not quite redundant with the infinity mode, but it's also not orthogonal. So I'm happy to either remove it, merge both options into a single option (maybe -r 0 can mean infinite?), or make some other changes.


By the way, I found out that it works by finding that the test_calendar actually fails:

$ ./_build/default/examples/calendar/test_calendar.exe -s 0
calendar: ....
calendar: PASS
$ ./_build/default/examples/calendar/test_calendar.exe -s 0 -r 50000
calendar: ....
calendar: FAIL

When given the input:

    [1899-08-31 14:14:45; 1900-07-10 00:40:53]

the test failed:

    1900-07-11 00:40:53 != 1900-07-10 00:40:53
raphael-proust commented 4 years ago

bump

raphael-proust commented 3 years ago

ping

stedolan commented 3 years ago

Merged! Thanks, and apologies for forgetting about this.