testing: fuzz tests not discovered unless package path specified or GOMAXPROCS=1

What version of Go are you using (`go version`)?

$ go version
go version devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (release meaning dev.fuzz)

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/okokes/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/okokes/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/okokes/sdk/gotip/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/okokes/git/smda3/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build934674902=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When running go test -v -fuzz with GOMAXPROCS > 1 and ./... target, fuzzing doesn't start, the test binary just hangs. If I drop the // +build gofuzzbeta tag, the fuzzing sometimes does start (but it is not flushed to stdout).

If I set GOMAXPROCS=1 or I specify the exact path of my fuzzed package, fuzzing starts almost immediately and all the logs flush every three seconds.

Here's a concrete reproducer. Sorry for using a pre-existing project - when I tried a minimal reproducer... it all worked all of a sudden.

# commit 7d9672fab55920d245a0b78636e4ba512c025930
git clone https://github.com/kokes/smda -b experiments/fuzzing
cd smda
# this hangs
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
# these two work
GOMAXPROCS=1 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./src/database

I even tried building go from source, introducing various fmt.Printlns around the codebase. It seemed like CoordinateFuzzing was never even triggered, but I couldn't figure out why.

To try and isolate as much of my local software as possible, I ran a clean golang Docker image, built Go's dev.fuzz from scratch and ran the test command than would hang on my Mac. It did the same thing on this linux/amd64 image, so I don't think it's specific to Darwin or my setup.

What did you expect to see?

Fuzzing stats from logStats

What did you see instead?

Nothing, go test hangs.

Thanks for reporting this. I believe this is a symptom of #46312: we don't yet have support for fuzzing multiple targets in multiple packages.

I tried this out, and go test did appear to hang. ps showed a couple of worker processes running on github.com/kokes/smda/src/database. I sent SIGINT to one of them, and the whole command wrapped up with normal output printed all at once:

...
fuzzing, elapsed: 36.0s, execs: 1556484 (43234/sec), workers: 2, interesting: 6
fuzzing, elapsed: 39.0s, execs: 1685642 (43220/sec), workers: 2, interesting: 6
fuzzing, elapsed: 42.0s, execs: 1817287 (43268/sec), workers: 2, interesting: 6
fuzzing, elapsed: 45.0s, execs: 1945143 (43221/sec), workers: 2, interesting: 6
fuzzing, elapsed: 47.8s, execs: 2053839 (42963/sec), workers: 2, interesting: 6
--- PASS: FuzzDelimiterInference (47.81s)
PASS
ok      github.com/kokes/smda/src/database  47.962s
testing: warning: no targets to fuzz
PASS
ok      github.com/kokes/smda/src/query 0.221s [no targets to fuzz]
...

So the problems here are at least:

One only target is actually being fuzzed.
We're not streaming log messages.

I think GOMAXPROCS=2 may have something to do with the output streaming, but I'm not sure what exactly. I think go test may not stream output when multiple packages are running concurrently.

cc @golang/fuzzing

golang / go