search

dvyukov / go-fuzz

Randomized testing for Go
Apache License 2.0
4.76k stars 276 forks source link

make fuzzing a first class citizen in Go #154

Open dvyukov opened 7 years ago

dvyukov commented 7 years ago

There is a proposal for making fuzzing a first class citizen in Go: https://github.com/golang/go/issues/19109 First they are trying to understand if there's interest. I would appreciate if you drop a line there if you found fuzzing useful and a brief of your success story. Thanks

DavidVorick commented 7 years ago

Yes, we've found fuzzing useful in our projects multiple times. Especially sensitive code, the fuzzer will frequently find edge cases that we missed.

I will say that most of the benefit is usually seen in the first tiny bit of fuzzing. There's a pretty strong diminishing returns as you continue to fuzz, at least that's what we've found.

dvyukov commented 7 years ago

@DavidVorick Please post this at golang/go#19109

dvyukov commented 7 years ago

FYI here is a detailed proposal: https://docs.google.com/document/u/1/d/1zXR-TFL3BfnceEAWytV8bnzB2Tfp6EPFinWVJ5V4QC8/pub If you have any comments, post them to https://github.com/golang/go/issues/19109

andybons commented 6 years ago

I’m actively working on this.

andybons commented 6 years ago

An update on progress:

I’ve been a bit busy with the China launch so this has taken a back seat to that. I’m going to try to page this stuff back into my brain this week to get a better understanding of the code and what is required.

As stated on golang/go#19109, the next step is to get go-fuzz the closest it can be to our desired API in the go command so that we can better evaluate it as a candidate in the go tool, so I’ll be working on that first. This will also help me get more familiar with what is needed if it’s integrated into the go tool.

dvyukov commented 6 years ago

There are 2 things about the current go-fuzz code:

  1. It's overly complex in some places.
  2. It's dirty in lots of places.

So I am thinking a good way forward can be: You create a new branch and start working on the API part from scratch first. Do something that implements the proposed API, clean and close to upstreamable. The fuzzing part can be as simple as "let's just give it a random byte slice". After this part it should already be usable and should be able to find trivial bugs. At this point we can integrate with OSS-Fuzz already. Then we can add coverage and very basic mutation logic. At this point we can upstream this. Then we probably will need to shake out some stuff. And later we can make fuzzing logic more complex incrementally.

This is incremental plan with several milestones. We will deliver early and solve the important problems early without burring in complexities of fuzzing logic. And you obviously can look at the existing code and take any parts you need. Otherwise I hardly see how we can progress from the current go-fuzz state. Thoughts?

andybons commented 6 years ago

Sounds good to me!

kcc commented 6 years ago

I would advocate for a slightly different order:

Then iterate.

AlekSi commented 6 years ago

One thing I would like to see in the integrated version is #65. In my experience, that is the biggest stopper for people trying to use go-fuzz.

dvyukov commented 6 years ago

Re #65, agree. The current state-of-art in fuzzing seems to be moving in this direction (libfuzzer's protobuf-based mutation, syzkaller). But we again can do the most awesome support for this across other languages as:

func Fuzz(f testing.F, ... arbitrary args here ...) {...}

The args can recursively include arbitrary types, including pointers, structs, slices (except for map/chan probably). Then we use reflect to capture the signature and store inputs along the lines of: {"abc", 0.57, {true, [0, 1, 2]}}.

guidovranken commented 5 years ago

Some time ago I appropriated @dvyukov 's Go instrumentation for usage with libFuzzer for my own needs: https://github.com/guidovranken/libfuzzer-go

I now have an oss-fuzz branch ready to be merged. It uses libFuzzer's extra counters for coverage. See: https://github.com/guidovranken/oss-fuzz/commit/0b48d4120731abaaf18d2722a749050c3f13706f

It now only features a basic JSON decoder but can easily be extended.

Are you interested in running this on oss-fuzz? If so, I'll make a PR, and transfer ownership to you (Golang team) once it's up. If not, no hard feelings.