search

google / clusterfuzz

Scalable fuzzing infrastructure.
https://google.github.io/clusterfuzz
Apache License 2.0
5.28k stars 553 forks source link

Don't file issues for crashes in custom mutators #2827

Open jonathanmetzman opened 1 year ago

jonathanmetzman commented 1 year ago

These are unreproducible and can be annoying to devs. This was requested by some CF users but I'm not sure we should do it. What do you think @oliverchang

afd commented 1 year ago

I'm one of the users who requested this. The motivation is to lower the risk of deploying adventurous, but potentially slightly unreliable, custom mutators. An adventurous custom mutator may lead to finding interesting bugs in the software under test, but if a lot of noise gets created by crashes in the adventurous custom mutator code being reported this can lead to wasted time, especially if the bugs in the "adventurous" code are low priority for fixing in their own right.

More generally what I believe would be useful is some support for distinguishing between crashes arising from custom mutator code vs. crashes arising from the code under test. Any of the following would be useful:

  1. Providing reproducible test cases for custom mutator crashes: when a crash comes from a custom mutator, provide as a test case the input buffer to the mutator and the random number generator seed that was passed to the mutator. Assuming at repro time the same random number generator implementation is used, this should allow the crash to be reproduced, which would make it possible to debug the custom mutator - this would be extremely useful.

  2. Automatically labelling filed issues to indicate that they come from a custom mutator. This would allow such issues to be ignored, if desired, during bug triage, by filtering them out based on their label.

  3. Having a mode whereby bugs in custom mutators are not filed at all. This is what I originally suggested, but if it's possible to easily filter out such bugs via (2) above, and furthermore to have help getting started in debugging them thanks to (1) above, I don't think there would be a use case for ignoring them altogether.

oliverchang commented 1 year ago

Hi there, it's important that we file such issues. Let me explain some of our rationale:

We typically do not file unreproducible crashes. In the case of https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40225, this was because this was repeatedly being hit, blocking fuzzing from doing useful work. If we don't file such crashes, fuzzers will silently not be very productive without developers knowing about it. It's critical that these issues are fixed.

Re reproducibility, the logs will contain things like INFO: Seed: 3708661885, and a testcase will be provided that caused the crash, but we aren't able to capture all the state (i.e. invocations of the rng) that led up to this so it may still end up being unreproducible locally.

I don't have a good solution to this that we can easily implement -- perhaps another solution to this is to write a fuzzer for the custom mutator itself to be able to pinpoint the issue? @afd

afd commented 1 year ago

Thanks @oliverchang - the idea of writing a fuzzer for the custom mutator is a good one.

In cases where a report is filed (and I take the point about fuzz blockers), I also wonder how hard it would be to automatically label the bug with something to indicate that the cause is a custom mutator (based on the stack trace of the crash).