feat: fuzz corpus saving and replay in standard format

mds1 commented 2 years ago

Component

Forge

Describe the feature you would like

https://github.com/foundry-rs/foundry/issues/991 was closed by https://github.com/foundry-rs/foundry/pull/1658, but as documented by this comment there's some other related work, so pulling that into separate issues.

Follows https://github.com/foundry-rs/foundry/issues/2551 to allow saving and replaying entire fuzz campaigns instead of just failures.

Capture fuzz campaigns via ResultCache (Or some other mechanism TBD).
Persist fuzz campaigns to /cache/fuzz/corpus-latest.json.
Re-use foundry.toml flag to specify outdir.
Add a replay proptest strategy that is powered by a corpus.json file.
Extend --fuzz-input-file option to additionally take a corpus.json file & forge test to trigger replay.
Ensure /cache/fuzz is removed with forge clean.
Consider a --generate-test-cases flag to autogenerate concrete tests from the fuzz inputs. these can be logged to the console for the user to put anywhere, or inserted directly below the failed fuzz test in the test file

Including the generated dictionary (perhaps as an optional flag if it's too big?) would be useful as well

Additional context

No response

aviggiano commented 10 months ago

Relevant discussion regarding the standardization of the corpus between fuzzing and property based testing tools:

https://github.com/crytic/medusa/issues/234

0xalpharush commented 5 months ago

I don't have a strong sense of what the right format would be but it would be nice to extend the export of https://github.com/foundry-rs/foundry/pull/7899 to include the traces of unit tests as well

grandizzy commented 5 months ago

I don't have a strong sense of what the right format would be but it would be nice to extend the export of #7899 to include the traces of unit tests as well

Good idea, I created #8114 to track this, please add ant other relevant comment there. thank you!

foundry-rs / foundry