google / gofuzz

Fuzz testing for go.
Apache License 2.0
1.5k stars 119 forks source link

go-fuzz integration may give nil values despite NilChance(0) #46

Closed ericcornelissen closed 4 years ago

ericcornelissen commented 4 years ago

I have not fully investigated this problem yet, but I figured I can share it already at this stage. I found that while using gofuzz (this project) with the new go-fuzz integration I can get nil values even if I use NilChance(0).

For a working example I will use the go-fuzz integration in a normal test. Following (roughly) one of the examples found in the README, I create a test where I fuzz fancyStruct using data that was generated for me by go-fuzz while setting NilChance to 0. Then I check if any of the struct fields is nil. You can see the code blow :arrowdown: Now, if you run this test you will (should?1) find that it fails and outputs "not good A"_.

Based on my experience so far this happens when the byte string is null-terminated (i.e. it ends with \x00). I don't know if it can happen 1) if \x00 appears elsewhere in the string (but in the example below it doesn't work with e.g. []byte("H000000\x000")), 2) if there are any null-terminated strings for which it doesn't happen, or 3) if other "special" characters can cause it. I also haven't look at the source code of this project to see what might be causing the problem.

package testing_something

import (
    "testing"

    fuzz "github.com/google/gofuzz"
)

func TestGoFuzz(t *testing.T) {
    data := []byte("H0000000\x00")
    f := fuzz.NewFromGoFuzz(data).NilChance(0) // Instead of f := fuzz.New().NilChance(.5)
    var fancyStruct struct {
        A, B, C, D *string
    }
    f.Fuzz(&fancyStruct) // None of the pointers should be nil, as NilChance is 0

    if fancyStruct.A == nil {
        t.Error("not good A")
    }
    if fancyStruct.B == nil {
        t.Error("not good B")
    }
    if fancyStruct.C == nil {
        t.Error("not good C")
    }
    if fancyStruct.D == nil {
        t.Error("not good D")
    }
}

  1. I'm guessing this may possibly be different per OS or golang version. In case that is true, I was running this in a golang docker container for v1.14 (so docker run [ARGUMENTS] golang:1.14) and installed just go-fuzz using go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
lavalamp commented 4 years ago

https://github.com/google/gofuzz/blob/master/fuzz.go#L175

My current suspicion is that rand.Float64() literally returned 0, which would be much more astonishing if your input weren't feeding it zeros :) If you can confirm, I'd support changing it to >= instead of > and adding your test as a new test case.

ericcornelissen commented 4 years ago

My current suspicion is that rand.Float64() literally returned 0, which would be much more astonishing if your input weren't feeding it zeros :) If you can confirm, I'd support changing it to >= instead of > and adding your test as a new test case.

I just debugged this and rand.Float64() is indeed returned 0, I will open a PR with a fix and a test case momentarily.