testing: unclear what t.Skip does in the context of fuzzing

[mvdan]

The current "first class fuzzing" proposal design doc (https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md) only has one mention of "skip":

    // Run the fuzz test
    f.Fuzz(func(t *testing.T, a string, num *big.Int) {
        t.Parallel() // seed corpus tests can run in parallel
        if num.Sign() <= 0 {
            t.Skip() // only test positive numbers
        }

As I was porting one of my fuzz funcs from go-fuzz to first class fuzzing, I remembered that go-fuzz actually has two kinds of "skips":

The function must return 1 if the fuzzer should increase priority of the given input during subsequent fuzzing (for example, the input is lexically correct and was parsed successfully); -1 if the input must not be added to corpus even if gives new coverage; and 0 otherwise; other values are reserved for future use.

So it seems like, with go-fuzz, you have three options: give priority (+1), must ignore (-1), or default behavior (0).

Personally, I only used +1 or 0 in my go-fuzz funcs; I never had any input that must never be added to the corpus. I've adapted those returns for the new fuzzer so that the return 1 is now a regular return ending the function, and return 0 is now a t.Skip to signal an uninteresting input. Here's an example: https://github.com/mvdan/sh/commit/e186e04cbf119ac6e51d2010cb37674374557046

I'd love it if the meaning of t.Skip in the context of fuzz funcs was better documented. Perhaps my understanding of what it does is incorrect, or perhaps it doesn't correspond to return 0 in go-fuzz.

Another question is whether we want an equivalent to go-fuzz's return -1. I personally didn't have a need for it, but others might be using it and not know how to transition those to the new fuzzer.

cc @katiehockman

[mvdan]

The docs do say:

The Skip method of *T can be used in a fuzz target if the input is invalid, but should not be considered a crash.

Still unclear to me if this means that the input can end up in the cached corpus inside the build cache or not - to my understanding, that would be the difference between go-fuzz's 0 and -1.

[jayconrod]

cc @golang/fuzzing We should clarify this.

Currently T.Skip does nothing specific to fuzzing; it's like returning early.

We should make sure that if T.Skip is called, an input will not be considered "interesting" even if it provides new coverage. I think that would be equivalent to return -1 in go-fuzz.

[mvdan]

So I guess go-fuzz's return 0 would be equivalent to just return without t.Skip? In that scenario I guess I'd mostly avoid t.Skip in the context of fuzzing, just like I didn't need to use return -1 for go-fuzz. Some guidance on when or when not to use Skip would be certainly welcome.

[rsc]

Did you mean Go 1.19 here? Or is this Go 1.18?

[jayconrod]

1.19 most likely, though it would be nice to have for 1.18. There are already a lot of open release blockers to fix soon though.

[ianlancetaylor]

CC @golang/fuzzing

What is the status of this issue? Thanks.

[gopherbot]

Change https://go.dev/cl/430676 mentions this issue: internal/fuzz: make T.Skip ignore inputs when fuzzing