golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.9k stars 17.65k forks source link

cmd/go: verify go test flags with -fuzz #49294

Open katiehockman opened 3 years ago

katiehockman commented 3 years ago

This is more of a tracking issue, just to have a record of how go test -fuzz behaves when also passed other flags. Let's make sure this is reasonable for the upcoming release for now, and see if we can determine what it would should look like in the long term.

For 1.18, the bare minimum is that it doesn't break anything :)

With -run the coverage is printed as zero. We can probably just document that it isn't supported.

godev test -fuzz Fuzz -run Fuzz -cover
...
PASS
coverage: 0.0% of statements
ok      crypto/dsa      3.551s

Without -run the coverage is shown for the other tests that were run. That's good, but I don't want people to get the impression that this coverage includes what was found during fuzzing. Maybe we should print a warning if -cover is provided with the -fuzz flag to indicate that the coverage % doesn't include anything from fuzzing.

godev test crypto/dsa -fuzz FuzzFoo -cover                     
...
PASS
coverage: 88.3% of statements
ok      crypto/dsa      29.526s

These simply don't affect fuzzing right now, and maybe never should. We should either document that, add a warning to the command line before fuzzing starts, or both.

Works as intended.

Not 100% sure if this is working as intended, but it at least isn't break anything.

bcmills commented 3 years ago

-short seems like it could affect fuzzing, right? (If a fuzz test calls the testing.Short() function, will it panic?)

It seems like -cpu could affect fuzzing too (by changing runtime.GOMAXPROCS(0)).

-failfast and -vet should already work in combination with fuzzing (should be completely orthogonal, IIUC).

-shuffle should probably shuffle the order of test inputs run from the existing corpus.