elastic / apm-data

apm-data holds definitions and code for manipulating Elastic APM data
Apache License 2.0
10 stars 25 forks source link

Introduce fuzz testing #66

Open kruskall opened 1 year ago

kruskall commented 1 year ago

Followup from https://github.com/elastic/apm-data/pull/63

We should introduce fuzz testing to make sure we are not missing anything

As this is a library we might also evaluate adding fuzz testing into APM Server and fuzz the intake endpoint directly.

kruskall commented 1 year ago

RE https://github.com/elastic/apm-data/pull/63#issuecomment-1595960376:

the stdlib does not support structured fuzzing or fuzzing complex types (structs). We need a compatibility layer to cover that usage. In that PR I used https://github.com/google/gofuzz.

Gofuzz has full support for the stdlib fuzzing engine and the test looks like this:

func FuzzError(f *testing.F) {
    f.Fuzz(func(t *testing.T, data []byte) {
        c := complexStruct{}
        fuzz.NewFromGoFuzz(data).NilChance(.1).NumElements(1, 10).MaxDepth(5).Fuzz(&c) // populate complex struct
        foo(&c) // run target method with fuzzed data 
    })
}

The stdlib is generating random data ([]byte) which are used as a seed to populate the complex struct (this is to make the tests reproducible).

I looked into https://github.com/AdaLogics/go-fuzz-headers but my understanding is that it is used for https://github.com/dvyukov/go-fuzz and not to wire complex structs with the stdlib engine.

axw commented 1 year ago

https://adalogics.com/blog/structure-aware-go-fuzzing-complex-types gives some examples of using go-fuzz-headers with Go's native fuzz testing.