feat: support time-based and continuous fuzzing and invariant testing

[mds1]

Component

Forge

Describe the feature you would like

Right now fuzz campaigns are defined by their number of runs. It’s also useful to instead define campaigns with a timeout which specifies the duration of the campaign, with a value of 0 meaning to run indefinitely. This is common in other fuzzers such as Echidna.

It would also be helpful to support config via cheat codes as discussed in #744, to allow some fuzz/invariant tests to be defined by number of runs with others defined by a timeout

Additional context

No response

[onbjerg]

Noting here that it does not seem like proptest supports time-based fuzzing out of the box, so we will have to roll our own - that does look like it is somewhat doable, though. We need to wrap the test runner in some time based executor and continually call TestRunner::run_one, but this function takes a case, not a strategy, so it looks like we might need to manage the simplification/complication of values etc. ourselves

[mds1]

Proptest is largely unmaintained at this point. There hasn't been an update for over a year, though there is a fork that's made a few fixes—more info here: https://github.com/AltSysrq/proptest/issues/268

I only mention that because two alternative options for implementation include:

1. Forking and maintaining proptest to add support for features like this
2. Switching to a better maintained fuzzing framework that has this feature implemented (or makes it easy to wrap around). A few alternatives to look into can be found in https://github.com/foundry-rs/foundry/issues/387, but a lot may have changed since that issue was created so there may be newer/better crates not listed

[onbjerg]

I think we should consider switching at some point. I looked around but didn't find anything that immediately struck me but I haven't looked much. I think forking and maintaining a fuzzer on top of Cast + Anvil + Forge + supporting stuff (foundry-hardhat, book etc.) is probably stretching resources a bit thin 😄 Will take a look at the ones linked in #387

[rex-remind101]

Your input on this thread would be appreciated since it sounds like you all have some direction already https://github.com/AltSysrq/proptest/issues/268#issuecomment-1240149041

[0xPhaze]

Any progress on this one?

[wuestholz]

I'm not familiar with the internals of how proptest is invoked, but I was wondering if the following approach could work: wrap the proptest invokation by repeatedly running proptest for a small number of runs until the time limit has been reached. Each "short run" needs to provide a different random seed (derived from the user-provided random seed) when invoking proptest and the results of all "short runs" need to be aggregated.

Feel free to ignore this if the proposed approach doesn't make sense. :)

[grandizzy]

proposed way to interrupt runs for fuzz / invariant tests when timer expires https://github.com/proptest-rs/proptest/issues/460 to be applied in addition to using rayon collect / short-circuit on Err