thanm
commented
3 years ago @jayconrod @katiehockman
Open stevenjohnstone opened 3 years ago
thanm
commented
3 years ago @jayconrod @katiehockman
toothrot
commented
3 years ago Checking in on this issue as it's labeled a release blocker for Go 1.18. Is there any update?
jayconrod
commented
3 years ago Not yet. There are a lot of fuzzing issues. We'll get to them as soon as we can.
katiehockman
commented
3 years ago I'm going to remove the release-blocker label for this issue. I think it's a nice to have, so I'll leave it for Go 1.18, but not strictly necessary for the release. We don't currently support any detection around what kind of crash occurred in order to replicate it during minimization. This is something that we'll need down the line: namely for deduplication of crashing so that we can run in a -fuzzcontinue mode that doesn't stop after the first crash.
In the meantime, it is still reporting a crash, even if it isn't the first one found. Once that crash is fixed, then the fuzzer should be able to find the other one as well.
stevenjohnstone
commented
2 years ago @katiehockman I think the problem here is that minimization is considered successful if the new value has at least one coverage bit in common with the original. This can lead to two inputs with little in common e.g for this code
if len(input) >= 1 {
if input[0] == 'A' {
foo(input);
} else {
bar(input);
}
}[]byte{'B'} could be considered a minimization of []byte{'A', 1, 2, 3} because they both make it into the enclosing if body and so have a coverage bit in common.
I think a minimized input should never result in fewer coverage bits than we had for the original input. I experimented with enforcing this and the following patch appears to fix the bug:
diff --git a/src/internal/fuzz/coverage.go b/src/internal/fuzz/coverage.go
index 3dee73b81c..88f98a16b2 100644
--- a/src/internal/fuzz/coverage.go
+++ b/src/internal/fuzz/coverage.go
@@ -66,6 +66,17 @@ func countNewCoverageBits(base, snapshot []byte) int {
return n
}
+// isCoverageSubset returns true if all the base coverage bits are set in
+// snapshot
+func isCoverageSubset(base, snapshot []byte) bool {
+ for i, v := range base {
+ if v&snapshot[i] != v {
+ return false
+ }
+ }
+ return true
+}
+
// hasCoverageBit returns true if snapshot has at least one bit set that is
// also set in base.
func hasCoverageBit(base, snapshot []byte) bool {
diff --git a/src/internal/fuzz/worker.go b/src/internal/fuzz/worker.go
index e7d824bea1..b80e98bde7 100644
--- a/src/internal/fuzz/worker.go
+++ b/src/internal/fuzz/worker.go
@@ -908,7 +908,7 @@ func (ws *workerServer) minimizeInput(ctx context.Context, vals []interface{}, c
}
return true
}
- if keepCoverage != nil && hasCoverageBit(keepCoverage, coverageSnapshot) {
+ if keepCoverage != nil && isCoverageSubset(keepCoverage, coverageSnapshot) {
return true
}Given recent improvements in the effectiveness of the mutator by @rolandshoemaker I've been able to see that this problem with minimization actually works against the mutator. For example, this problem renders the mutator ineffective for examples which libfuzzer etc deal with easily just using coverage feedback e.g. the fuzzer can't find an input which terminates the following
var leet = []byte{1, 3, 3, 7}
func magic(input []byte) bool {
if len(input) >= 4 && input[0] == leet[0] && input[1] == leet[1] && input[2] == leet[2] && input[3] == leet[3] {
input = input[4:]
return len(input) >= 4 && input[0] == leet[0] && input[1] == leet[1] && input[2] == leet[2] && input[3] == leet[3]
}
return false
}
func Fuzz(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
if magic(data) {
t.Fatalf("magic is %v", data)
}
})
}With the above patch applied, the fuzzer finds a crasher right away. My interpretation is that the mutator is working to increase coverage but minimization can reduce coverage.
katiehockman
commented
2 years ago Thanks @stevenjohnstone for looking into this so deeply. One of us should be able to take a look at this in the next couple weeks as well, and hopefully can improve this before 1.18. If not, it'll happen for 1.19.
gopherbot
commented
2 years ago Change https://go.dev/cl/391454 mentions this issue: internal/fuzz: minimization should not reduce coverage
gopherbot
commented
2 years ago Change https://go.dev/cl/391614 mentions this issue: internal/fuzz: don't use dirty coverage maps during minimization
gopherbot
commented
2 years ago Change https://go.dev/cl/391797 mentions this issue: [release-branch.go1.18] internal/fuzz: don't use dirty coverage maps during minimization
gopherbot
commented
2 years ago Change https://go.dev/cl/391798 mentions this issue: [release-branch.go1.18] internal/fuzz: minimization should not reduce coverage
ianlancetaylor
commented
2 years ago CC @golang/fuzzing
Looks like this didn't make 1.19. Moving to backlog. Please recategorize as appropriate. Thanks.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
n/a
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Here's the simplest case I can find which reproduces the problem:
The test is rigged so it'll crash in
branch1. Minimization kicks in and it ends up crashing inbranch2:Instrumenting with this bpftrace script
I see that all the worker processes run
branch1first:The tail of the log shows what looks like the negative value
-20123being cast to a largeuintand then divided by 10 repeatedly to reach the value 18446 in the corpus. This may be another issue if the intention was to preserve the sign of integers on minimization :shrug:What did you expect to see?
I expected to see the original crash recorded. The first and second crashes have different coverage and different panic messages/backtraces. There's enough information to label these as different crashes.
In a more complicated example, reducing one type of crash to another and then discarding the original wastes a lot of work. The fuzzer would need to be run again to find the original crash. It'd be really useful if the fuzzer retained the original crasher and provided a minimized version as a feature.
What did you see instead?
The original crasher is lost.