stedolan / crowbar

Property fuzzing for OCaml
MIT License
183 stars 28 forks source link

decreased stability in xmldiff example with abstract gens API #14

Closed yomimono closed 3 years ago

yomimono commented 6 years ago

Fuzzing xmldiff in cffd7df0ab1daa9109220c329eba871c0759db98 gets 100% reported stability in afl-fuzz. Fuzzing xmldiff in 85401097ee3b125048dea36a613b885e1664d7bb reports stability that fluctuates in the high 99%s.

I wouldn't expect for them to be different?

yomimono commented 6 years ago

And also some exciting new results in the stdlib tests!

stedolan commented 6 years ago

I think I've figured this one out, but it will require an OCaml runtime patch to fix.

Lazy values are implemented as something not entirely unlike this:

type 'a lazy_contents = Lazy_tag of (unit -> 'a) | Forward_tag of 'a
type 'a lazy = lazy_contents ref

Once the value is forced, it's turned into Forward_tag. The GC has an optimisation where it removes the indirection at runtime, turning Forward_tag a into just a.

The code in Lazy.force has a branch to check this: if it's Lazy_tag, it forces, if it's Forward_tag, it returns the forwarded value, and if it's anything else it returns the value directly, assuming the GC optimisation kicked in.

The trouble is that that last branch is not significant. afl-fuzz is picking up on a different codepath depending on whether the GC ran or not, but this never makes a difference to the semantics of the program so it should be ignored. It's the same type of problem as the one @talex5 found with object end, but this one should be much easier to fix because the Lazy code is much simpler than class creation.

stedolan commented 6 years ago

I think this should be fixed by ocaml/ocaml#1754