Closed capnspacehook closed 9 months ago
(CC @rolandshoemaker)
I've also noticed this in one test; after running with 0/sec for a while (but clearly not actually 0, given my computer's CPU usage) it came back with an unusually, almost implausibly, high number of execs/sec for a few runs before returning to normal:
fuzz: elapsed: 0s, gathering baseline coverage: 0/9 completed
fuzz: elapsed: 0s, gathering baseline coverage: 9/9 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 79681 (26558/sec), new interesting: 35 (total: 44)
fuzz: elapsed: 6s, execs: 93444 (4588/sec), new interesting: 36 (total: 45)
fuzz: elapsed: 9s, execs: 153249 (19933/sec), new interesting: 39 (total: 48)
fuzz: elapsed: 12s, execs: 166674 (4475/sec), new interesting: 40 (total: 49)
fuzz: elapsed: 15s, execs: 438829 (90711/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 18s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 21s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 24s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 27s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 30s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 33s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 36s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 39s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 42s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 45s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 48s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 51s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 54s, execs: 438829 (0/sec), new interesting: 41 (total: 50)
fuzz: elapsed: 57s, execs: 2051825 (537812/sec), new interesting: 45 (total: 54)
fuzz: elapsed: 1m0s, execs: 3173094 (373707/sec), new interesting: 49 (total: 58)
fuzz: elapsed: 1m3s, execs: 3257748 (28217/sec), new interesting: 50 (total: 59)
fuzz: elapsed: 1m6s, execs: 3450825 (64365/sec), new interesting: 52 (total: 61)
fuzz: elapsed: 1m9s, execs: 4098105 (215768/sec), new interesting: 52 (total: 61)
fuzz: elapsed: 1m12s, execs: 5313498 (405066/sec), new interesting: 54 (total: 63)
fuzz: elapsed: 1m15s, execs: 5483457 (56668/sec), new interesting: 55 (total: 64)
fuzz: elapsed: 1m18s, execs: 5647501 (54672/sec), new interesting: 57 (total: 66)
fuzz: elapsed: 1m21s, execs: 5834412 (62312/sec), new interesting: 57 (total: 66)
fuzz: elapsed: 1m24s, execs: 6009865 (58477/sec), new interesting: 57 (total: 66)
fuzz: elapsed: 1m27s, execs: 6185876 (58669/sec), new interesting: 57 (total: 66)
fuzz: elapsed: 1m30s, execs: 6362102 (58654/sec), new interesting: 57 (total: 66)
CC @golang/fuzzing
What is the current status here? This issue is currently in the 1.19 milestone. Should it move to 1.20? To Backlog? Thanks.
I can still easily reproduce this in github actions, it doesn't seem to be as specific as I thought: https://github.com/capnspacehook/egress-eddie/actions/runs/3374243584/jobs/5599662068
There is at least one instance of this in the Go build dashboard as well: https://build.golang.org/log/99d38c470b4df7a559bb833dc8f1fd15bf8d1aa3
I'm seeing this happen too (i.e., 10-30 second stretches of 0 execs/sec reported). I've instrumented my fuzz target to call T.Fatal if it takes longer than 1 second to finish, and it doesn't seem to trigger at all. So I'm guessing it's just a UI issue, and that the fuzzer is actually steadily working the whole time still.
@mdempsky, I don't think it's just a UI issue, or the cmd/go
test that runs a fixed number of times would have terminated. It may be an accounting issue, or a deadlock (or livelock) in which none of the workers are able to start new work.
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Change https://go.dev/cl/475196 mentions this issue: internal/fuzz: avoid deadlock on duplicate entris with exec limit
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Looks like CL 475196 didn't quite fix it: commit e671fe0c is after it on the main branch.
Looks like we switched from test_fuzz_mutator
triggering it to test_fuzz_cov
triggering it. Either another deadlock, or I just managed to shuffle it around...
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Change https://go.dev/cl/476595 mentions this issue: internal/fuzz: improve debugging messages
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Change https://go.dev/cl/476815 mentions this issue: internal/fuzz: release lock when reading file fails
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
@bcmills I think we might've also squashed this 🎉.
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),`
Commit 023b542e is on release-branch.go1.20
(#59483).
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "cmd/go" && `fuzz: elapsed: \d+m\d+s, execs: \d+ \(0/sec\),` && (date < "2023-10-01" || builder ~ `^go1.20`)
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
invalid_utf8
branchgo_fuzzer_bug
branchfuzzcache
replace
directive to thego.mod
file replacinggithub.com/BurntSushi/toml
with where you clonedgithub.com/capnspacehook/toml
go test -run Parse -fuzz Fuzz
What did you expect to see?
The fuzzer trying many inputs per second
What did you see instead?
May be related to #48591.