search

stedolan / crowbar

Property fuzzing for OCaml
MIT License
183 stars 28 forks source link

try slightly harder to fix nonterminating size policy #21

Closed yomimono closed 6 years ago

yomimono commented 6 years ago

This is a rebase and continuation of #18 .

You can see this branch in action on the ocaml-test-stdlib tests here, where the tests running in a 4.05.0 environment fail with a stack overflow but the tests in 4.06.0 succeed (meaning they run for 25 minutes under afl-fuzz without finding any crashes).

I've generally been in a 4.05.0 universe when using ocaml-test-omp, which may explain (shallowly - I don't understand why 4.05.0's performance would be dramatically worse) why various fixes worked for @stedolan but not me.

stedolan commented 6 years ago

I've generally been in a 4.05.0 universe when using ocaml-test-omp, which may explain (shallowly - I don't understand why 4.05.0's performance would be dramatically worse) why various fixes worked for @stedolan but not me.

I think a better explanation is that you did more thorough testing! My patch made failures under the 5000-run mode go away, but I didn't test more than that.

stedolan commented 6 years ago

Merged because this is definitely an improvement, but this size stuff is getting quite messy. The trick of pulling the constants out of choose was always a hack, and while this patch makes the hack more robust it doesn't make it less hacky.

I'm considering changing the core to represent the type 'a gen as something like:

{ generate : int -> state -> 'a; small_examples : 'a list }

That way, every generator comes with a few small examples which are constructed when the generator is built (i.e. once per fuzzing run, before any input has arrived), and used when the size is sufficiently small. If a generator has no small examples, we'll get a failure much earlier rather than waiting for afl to find the stack overflow. Thoughts?

yomimono commented 6 years ago

I think that's a better idea than I've been able to come up with, for sure.