Viewing actions from fuzzed failure

[brunokim]

First of all, thanks for a great library! I just updated from v0.4.8 to v1.0.0, so I could use the fuzzing capabilities.

However, in my first attempt I've got a quite unreadable result, and would like some help on how to obtain the list of actions as they usually appear on failure.

--- FAIL: FuzzMultipleRunesProperty (61.81s)
    --- FAIL: FuzzMultipleRunesProperty (0.00s)
        value.go:586: [rapid] failed: can't find a valid action

    Failing input written to testdata/fuzz/FuzzMultipleRunesProperty/51c85e8f6597f491
    To re-run:
    go test -run=FuzzMultipleRunesProperty/51c85e8f6597f491

The affected code is available at https://github.com/crdteam/causal-tree/pull/12. I've created a rather complex state machine and a regular test passes successfully, but the fuzzed one fails:

func TestMultipleRunesProperty(t *testing.T) {
    rapid.Check(t, func(t *rapid.T) {
        t.Repeat(rapid.StateMachineActions(newMultipleRunesModel()))
    })
}

func FuzzMultipleRunesProperty(f *testing.F) {
    f.Fuzz(rapid.MakeFuzz(func(t *rapid.T) {
        t.Repeat(rapid.StateMachineActions(newMultipleRunesModel()))
    }))
}

[flyingmutant]

Should be fixed now! The can't find a valid action message means the state machine got into a place where the runner can't find any action that is not skipped.

I am curious though: if you run rapid with a really high number of checks (e.g. go test -rapid.checks=1_000_000 ...), will it find an error that the fuzzer has found in comparable time?

[brunokim]

Do you have any tips how can I view the sequence of steps until it reached this error?

[flyingmutant]

If you will update to using the latest rapid version from the master branch, you'll see the sequence without any changes from your side.

[brunokim]

The fuzzer found a failing condition just because the same (invalid) action is selected multiple times. Running with 1M checks found nothing :raised_hands:

$ go test ./...
?       github.com/brunokim/causal-tree/cmd/demo    [no test files]
--- FAIL: FuzzMultipleRunesProperty (0.00s)
    --- FAIL: FuzzMultipleRunesProperty/51c85e8f6597f491 (0.00s)
        ctree_property_test.go:196: [rapid] draw action: "DeleteString"
        ctree_property_test.go:109: no strings
        ...
        ctree_property_test.go:196: [rapid] draw action: "DeleteString"
        ctree_property_test.go:109: no strings
        value.go:586: [rapid] failed: can't find a valid (non-skipped) action
FAIL
FAIL    github.com/brunokim/causal-tree/crdt    0.060s
ok      github.com/brunokim/causal-tree/diff    (cached)
FAIL
$ go test github.com/brunokim/causal-tree/crdt -rapid.checks=100_000 -run='^Test'
ok      github.com/brunokim/causal-tree/crdt    37.838s
$ go test github.com/brunokim/causal-tree/crdt -rapid.checks=1_000_000 -run='^Test'
ok      github.com/brunokim/causal-tree/crdt    411.167s

How can I see Rapid test flags? I'd like to increase the limit for skipped actions to a ridiculous amount and see if the fuzzer let go of it.

[brunokim]

Ah, the parameter that controls the limit is not a flag:

https://github.com/flyingmutant/rapid/blob/master/statemachine.go#L17

I'll try changing this to a ridiculous value and see if the fuzzer keeps finding it as a failure mode. If I succeed in deceiving the fuzzer, I'll transform this "hack" into a flag.

[flyingmutant]

I am afraid that the fuzzer is just stuck choosing the same action over and over again, e.g. because fuzzer input is all-zero.

When running under the fuzzer, it can make sense to try to use a different strategy for choosing actions that does not rely on input being random.