stedolan / crowbar

Property fuzzing for OCaml
MIT License
180 stars 31 forks source link

Failing test not reported as crash #28

Closed copy closed 6 years ago

copy commented 6 years ago

Using a slightly modified version of the identity example:

let identity x = Crowbar.check_eq x (x+1)
let () = Crowbar.(add_test ~name:"identity function" [int] (fun i -> identity i))

Running the test without afl fails as expected:

% ./_build/default/test.exe
identity function: ....
identity function: FAIL

When given the input:

    -2701223470281276764

the test failed:

    different

But running through afl doesn't report any crashes, even after several cycles:

+- process timing -------------------------------------+- overall results -----+
|        run time : 0 days, 0 hrs, 0 min, 6 sec        |  cycles done : 76     |
|   last new path : 0 days, 0 hrs, 0 min, 5 sec        |  total paths : 2      |
| last uniq crash : none seen yet                      | uniq crashes : 0      |
|  last uniq hang : none seen yet                      |   uniq hangs : 0      |

This is on 4.06.0+afl using the master version of crowbar and afl-fuzz 2.52b.

stedolan commented 6 years ago

Thanks for the report! Should be fixed now. (The issue was that a recent refactor of the command-line handling accidentally swallowed the exception that Crowbar uses to inform afl-fuzz of test failures).