search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.39k stars 2.21k forks source link

Is OSS-fuzz wasting its time? #12006

Closed cristiandaniele closed 4 months ago

cristiandaniele commented 4 months ago

I analysed the nature of the SUT OSS-fuzz fuzzes. I passed through about 400 case studies and noted that almost 10% are stateful.

Since OSS only has stateless fuzzers, how is it useful to employ such a fuzzer to do stateful fuzzing?

I zoomed in on two different case studies, i.e. ProFTPD and open62541 and noted that the accuracy of the fuzzing targets really makes the difference when dealing with stateful systems. In fact, that's the only way to get rid of the statefulness of systems and then be able to use stateless fuzzers for stateful systems. It's worthwhile to note that fuzzing targets won't allow the reproducibility of almost any bugs, as it won't allow to reconstruction of the whole trace but only provides the last message that triggers the bug.

What about introducing a check in the framework that allows a stateful bunch of fuzzers to play a role when detecting a stateful SUT?

DavidKorczynski commented 4 months ago

I'm not entirely sure what you're asking. I'm sure there are areas where some harnesses are not performing optimal due to the target being stateful.

However, what's your specific ask here

What about introducing a check in the framework that allows a stateful bunch of fuzzers to play a role when detecting a stateful SUT?

Are you suggesting adding a new fuzzing engine, or what is a "check" in this context? If this is part of the fuzz engine itself, how about submitting it to e.g. AFL++ as a patch? You could also verify the improvements by way of https://github.com/google/fuzzbench

cristiandaniele commented 4 months ago

Exactly! It would be nice to add a bunch of stateful fuzzers (e.g. AFLNet, SGFuzzer etc) in addition to the stateless fuzzer already used by OSS. A very simple "check" might be the presence of "network system calls". I expect most of the programs that work over the network to be stateful as they might (?) implement network protocols.

AFL(++) has been proven not very effective in fuzzing stateful fuzzers, I believe something like AFLNet would fit it better. And yet yes, using (Pro)FuzzBench to monitor the effectiveness would be possible.

DavidKorczynski commented 4 months ago

A very simple "check" might be the presence of "network system calls". I expect most of the programs that work over the network to be stateful as they might (?) implement network protocols.

I think it's a small minority of harnesses in OSS-Fuzz that falls into this bucket fwiw. But I'm sure there are other stateful harnesses across OSS-Fuzz as well.

And yet yes, using (Pro)FuzzBench to monitor the effectiveness would be possible.

If this comes out showing there is a lot of impact then I would personally be positive towards adding this in OSS-Fuzz

cristiandaniele commented 4 months ago

Thank you very much for the reply! I'll reach out again once I have more concrete results!