search

bazel-contrib / rules_fuzzing

Bazel Starlark extensions for defining fuzz tests in Bazel projects
Apache License 2.0
87 stars 20 forks source link

Implement Python support (Atheris) #162

Open fmeum opened 3 years ago

fmeum commented 3 years ago

If there is interest, I could add py_fuzz_test backed by Atheris after the release for java_fuzz_test is done.

As far as I can tell, this would require the following special steps for local Python fuzzing, everything else would be mostly analogous to the integration of Jazzer:

  1. Generalize the existing logic for locating ubsan_standalone_cxx to arbitrary sanitizer libraries and extract it into a separate repository rule.
  2. Add a rule that prevents a given target from receiving fuzzing instrumentation, even if a fuzzing binary depends on it. This is needed since the native Atheris fuzzer library also contains helper functions for fuzz targets and thus will be depended on by fuzz tests, but should not be instrumented for fuzzing for optimal performance. This is a bit tricky, but can for example be achieved by a transition that undoes the fuzzing_binary_transition.
  3. Produce Bazel BUILD files for Atheris. That is actually quite simple (using 1. and 2.), but requires a choice between a) depending on clang >= 12 and b) building libFuzzer from source.

OSS-Fuzz support for Python then requires a choice between a) adding the CPython installation compiled in base-builder also to base-runner and b) crafting a packaging rule for py_binary backed by pyinstaller or a similar tool that can bundle a Python target with a Python runtime (--build_python_zip may do the job).

stefanbucur commented 3 years ago

I think that would be nice in principle, but I would like to first see adoption (a few users) of the java_fuzz_test rule and also a set of concrete users that would adopt py_fuzz_test if we get to add it here.

fmeum commented 3 years ago

When you say adoption, do you mean of local fuzzing or of the OSS-Fuzz support? I don't know of many non-Google Java-based OSS projects using Bazel to begin with, so the OSS-Fuzz support built into java_fuzz_test is probably more of a mid- to long-term investment.

On the other hand, local fuzzing support for Java (and potentially Python) for any, also closed source Bazel project is already very useful and was my primary motivation for adding the Jazzer integration. It is however more difficult to measure adoption for this use case.

stefanbucur commented 3 years ago

When you say adoption, do you mean of local fuzzing or of the OSS-Fuzz support?

Either - basically any Bazel project defining {java,py}_fuzz_tests in their codebase.

I don't know of many non-Google Java-based OSS projects using Bazel to begin with, so the OSS-Fuzz support built into java_fuzz_test is probably more of a mid- to long-term investment.

I believe it is important for the code to have real users, if only a handful. Otherwise, we essentially pay the engineering cost (added complexity, refactorings, etc.) while the code never runs in "real-life" to reap benefits.

This was the approach when developing the C++ rules, too: Envoy is a very large early customer that was on board when we started developing the project.

On the other hand, local fuzzing support for Java (and potentially Python) for any, also closed source Bazel project is already very useful and was my primary motivation for adding the Jazzer integration. It is however more difficult to measure adoption for this use case.

I agree with this - local fuzzing support has always been a key feature of the project. But ultimately the proof of usefulness stays in actual usage, so this is why I'd like to make sure the rules end up being used (by Google or non-Google projects, for local fuzzing or OSS-Fuzz - any kind of use).

fmeum commented 3 years ago

As a first step, I will mention rules_fuzzing support prominently in the Jazzer docs once java_fuzz_test has been released. Users have been asking for Bazel rule support, but of course I don't know whether their projects are public.