testing: visualize fuzz corpus coverage

[rolandshoemaker]

It would be nice to be able to visualize the source coverage provided by an input corpus, in a similar manner to the coverage information provided for unit tests.

Fuzzing counters are implemented in a slightly different fashion to the (testing) coverage counters, but probably a similar method can be used to produce compatible coverage profiles for use with go tool cover.

[rolandshoemaker]

This may be more complicated than it seemed on the surface.

Since we add the libfuzzer-style counters during compilation, while doing other things to the source (like in-lining), the code we are instrumenting does not necessarily look like the source on disk, making mapping from counter positions to source positions complicated.

One other approach would be to just use the existing go tool cover coverage instrumentation, in addition to the libfuzzer instrumentation, since it already exists. There may be downsides I cannot think of at the moment, but it seems like a rather simple approach.

[timothy-king]

One other approach would be to just use the existing go tool cover coverage instrumentation, in addition to the libfuzzer instrumentation, since it already exists. There may be downsides I cannot think of at the moment, but it seems like a rather simple approach.

IIUC the suggestion is to instrument the program with two different coverage metrics, produce a single binary where both coverage metrics are aggregated during execution, but the two different metrics are post processed differently. So go tool cover is used for producing reports and the libfuzzer style counters are used for determining if coverage increased. Is this an accurate understanding of the approach?

Another approach could be to re-run the fuzzer generated tests under a go tool cover instrumented binary to produce the coverage information. FWIW other fuzzing systems have accepted a similar division of labor at the cost of needing to compile 2 binaries.

[cagedmantis]

/cc @golang/fuzzing

[firelizzard18]

Another approach could be to re-run the fuzzer generated tests under a go tool cover instrumented binary to produce the coverage information. FWIW other fuzzing systems have accepted a similar division of labor at the cost of needing to compile 2 binaries.

It seems like this could be done with a minimal change: add a -fuzztestdata <dir> flag to go test -fuzz=FuzzFoo ./pkg that tells the fuzzer to load the generated corpus from <dir> instead of ./pkg/testdata/fuzz. Then users could do it themselves with go test -run=FuzzFoo -fuzztestdata $GOCACHE/fuzz/example.com/module/pkg -cover.

[randall77]

I just ran into wanting this.

% go test -fuzz=MyTest -coverprofile=cov.out .
cannot use -coverprofile flag with -fuzz flag

It would be nice if it were that simple.

I'd settle for some way to output the generated corpus, after a -fuzztime=, so that I could copy it into testdata/fuzz somehow. Then a normal test the next time around, with -coverprofile, would touch what the fuzzer was touching.

[ericchiang]

Apologies if this redundant, but I wanted to look at the fuzzing coverage for a project today and found this issue.

I ran the following commands as a workaround to copy the fuzz cache to "testdata/fuzz" and generate a cover profile:

mkdir -p testdata/fuzz
cp -r "$( go env GOCACHE )/fuzz/$( go list )/" testdata/fuzz
go test -coverprofile=coverage.out
go tool cover -html=coverage.out

which seems to work today

[0xalpharush]

The above is not as convenient for workspaces. It'd be nice to add this as a subcommand and also for the fuzzer to report where the corpus is being loaded from and stored to on disk. See also great suggestions in https://github.com/golang/go/issues/50432#issuecomment-1486676539