camshaft / bolero

property testing and verification front-end for Rust
https://camshaft.github.io/bolero
MIT License
181 stars 14 forks source link

`cargo bolero repro` command? #114

Open Ekleog opened 1 year ago

Ekleog commented 1 year ago

I can't find a way to repro a single fuzzer crash from the cargo bolero command line.

Am I missing something, or should we add such a way, eg. cargo bolero repro [fuzzer] [crash-artifact]?

Currently, the best I found is to use cargo test: it compiles much faster and starts by testing the fuzzer crashes, but even it actually tests all the fuzzer crashes and corpus, and not just the one I'd like to test; which slows down the edit-compile-test cycle

camshaft commented 1 year ago

I usually just use cargo test [testname] for this workflow. This is most likely the fastest way possible since we'd have to run cargo test --no-run in the cargo bolero command anyway.

It does mean that if you get a sanitizer failure, it likely won't fail inside cargo test. For that, you can use cargo bolero test [testname] --release false and compilation should be a bit quicker. The fuzzers do generally run the crashes first as well.

Ekleog commented 1 year ago

Hmm so what you're saying makes sense for a development point of view when someone already knows of how cargo bolero works. (well, and now that #115 landed, as before that libfuzzer fuzzers didn't actually start by running the crashes, and with a 30k corpus at 100exec/s it's noticeable :sweat_smile: )

However, if someone doesn't know of how bolero works, having a command to "do whatever but rerun the crash in this file" would make sense I think. For instance, if this person downloaded the crash file from a fuzzer infra like oss-fuzz (though this is pending on #98 for now, but let's assume it's already possible), and doesn't even have the corpus/ and crashes/ folders locally, being able to cargo bolero repro -p [package] [test] [artifact] and that'd run like cargo bolero test -p [package] [test] --release false --engine-args [artifact] (or similar for non-libfuzzer harnesses) would make sense I think.

Would you be open to a PR adding such a subcommand? :)