search

HypothesisWorks / hypothesis

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
https://hypothesis.works
Other
7.4k stars 578 forks source link

Implement an Atheris fuzzing backend #3912

Open Zac-HD opened 3 months ago

Zac-HD commented 3 months ago

Now that we support alternative backends, it'd be great to improve our support for fuzzing with https://github.com/google/atheris. Currently, you can use this with test_fn.hypothesis.fuzz_one_input(), but as for Crosshair the complicated parsing logic is difficult for a fuzzer.

It'd be great to manage the setup and fuzz calls, sensible defaults for instrumentation with the option of user control, using FuzzedDataProvider to implement a PrimitiveProvider. For bonus points, warm-start Atheris by loading from the Hypothesis database (inc. HypoFuzz's keys) and writing out the corpus translated into the FuzzedDataProvider format (difficulty: really annoying until we finish the IR project).

Not sure exactly what interface I'd want, whether it's just a backend and the setup is users' problem, or whether we also provide a test_fn.hypothesis.fuzz_with_atheris(...) helper function. It'd be nice, but require a lot of UX choices we might not want to make.

Zac-HD commented 2 months ago

See also: https://github.com/Zac-HD/hypofuzz/issues/36 for a HypoFuzz backend. That's likely to perform somewhat better for pure-Python code, but doesn't have the native support for native-code coverage.