golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
121.55k stars 17.41k forks source link

proposal: testing: add `testing.F.Corpus` to help with cross tests conversion #65253

Open Jorropo opened 6 months ago

Jorropo commented 6 months ago

Proposal Details

I have two fuzz tests which test the same code, however one of them accepts binary data and the second one takes strings (my code has an ASCII and binary representation).

I was able to find new issues by seeding the corpus from the other test by converting the representation.

So I propose this to be added:

// Corpus let you fetch the corpus from an other test, callback will be called with all the entry in the other test corpus.
// source must be a function pointer to the actual other test.
// callback must be a function pointer which has the same signature as the other test [F.Add] argument.
// Corpus must not be called once [F.Fuzz] has been called.
func (*F) Corpus(source, callback any)

Expected usage:

func FuzzString(f *testing.F) {
 for _, v := range testcase {
  _, err := decodeString(v)
  if err != nil { f.Fatal(err) }
  f.Add(v)
 }
 f.Corpus(FuzzBinary, func(_ *testing.T, b []byte) {
  v, err := decodeBinary(b)
  if err != nil { return }
  str, err := v.String()
  if err != nil { return }
  f.Add(str)
 })
 f.Fuzz(func(_ *testing.T, s string) {
  v, err := decodeString(s)
  if err != nil { return }
  testStuff(v)
 })
}

func FuzzBinary(f *testing.F) {
 for _, v := range testcase {
  b, err := decodeString(v)
  if err != nil { f.Fatal(err) }
  f.Add(b)
 }
 f.Corpus(FuzzString, func(_ *testing.T, s string) {
  v, err := decodeString(s)
  if err != nil { return }
  f.Add(v)
 })
 f.Fuzz(func(_ *testing.T, b []byte) {
  v, err := decodeBinary(b)
  if err != nil { return }
  testStuff(v)
 })
}

It's unclear to me if callback should be retained, if it is retained we could run multiple fuzz tests in parallel and convert entries from one into one from the other on the fly (it would require some tricky-ish synchronisation and validation internally). I think for now this is not needed but we can add a comment hinting this might be a thing in the future. I don't think this would break anyone since you can't currently run more than one fuzz test together.

Jorropo commented 6 months ago

I think it should also be legal to fetch your own corpus (altho a bit pointless). This is so if we have 10 interconnected fuzz test we don't need 10 snowflake fetch conversion. I could have a single corpus(*testing.F) function which fetch and add the corpus from all the 10 tests.

ianlancetaylor commented 5 months ago

CC @golang/fuzzing