search

stedolan / crowbar

Property fuzzing for OCaml
MIT License
183 stars 28 forks source link

avoid `Premature end of file` #8

Open bobot opened 7 years ago

bobot commented 7 years ago

Perhaps I'm not using correctly crowbar, but it seems that creating a correct test case is not user friendly: an empty file is not enough and I don't know if depending on the test cases you have to create a file arbitrary big. Instead of premature end of file couldn't crowbar consider that it is filled with null bytes?

PS: the makefile in examples should have examples that uses afl-fuzz even if they are not run by default. Except that crowbar is great!

yomimono commented 6 years ago

@bobot, did you run into this problem when running a Crowbar test under afl-fuzz? I had a problem where afl-fuzz would report one unique crash after about 3,500 runs, but when I reran the test program with the crash input, I would get premature end of file and the program would exit normally. The actual crash was caused by something else, unrelated to the file length, which crowbar and afl-fuzz were handling correctly, but which didn't turn up in a single run.

I was able to fix it in my environment by changing a constant in afl-persistent. Could you try pinning afl-persistent with this change (opam pin add afl-persistent https://github.com/yomimono/ocaml-afl-persistent.git#persist-less) and see whether you get more sensible behavior?

NathanReb commented 5 years ago

We recently ran into this with @pascutto. This works well with afl-fuzz which doesn't consider that as a crash but it can be slightly disturbing when debugging other issues.

I think a slightly improved error message in this case advising the user to provide larger input would help!

I'm happy to provide a patch for that!

As a side note, I wonder how AFL behaves in those cases and if reading 0x00 bytes once we reached the end of the input —as @bobot suggests— would make the fuzzing more efficient.

stedolan commented 4 years ago

A better error message is a good idea.

Reading 0x00 bytes won't help, though. Sometimes an arbitrarily long sequence of null bytes is a valid prefix of a test case, so silently supplying nulls will turn a failure into nontermination.

bobot commented 4 years ago

Could we generate a random sequence of bytes in this case, the probability of non termination would be smaller.

stedolan commented 4 years ago

No, often the probability of non-termination is high even for a random sequence of bytes. (See section 3.2 of Claessen & Hughes' original QuickCheck paper)