search

foundry-rs / foundry

Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust.
https://getfoundry.sh
Apache License 2.0
8.33k stars 1.76k forks source link

feat: add real-time counter for fuzz/invariant runs #585

Closed mds1 closed 5 months ago

mds1 commented 2 years ago

dapptools has a counter like the one in the screenshot below so you can track progress in fuzz runs while those fuzz runs are executing. I recently ran a 10k fuzz run test that uses ffi, so it was quite slow and appears as if tests hang. A counter that updates live like the one below would be helpful for monitoring fuzz/invariant run progress and having confidence tests are still running as expected

image

pbshgthm commented 2 years ago

I could see the run count, along with mean and median too - implemented in this line.

https://github.com/gakonst/foundry/blob/9de25d1928da362cd3b185ebaccc0e3d00aee2c8/forge/src/runner.rs#L101

Am I missing something here, or this feature is already implemented?

Screenshot 2022-01-26 at 1 54 42 PM
onbjerg commented 2 years ago

Oh, you are right.

Perhaps the request is probably moreso the real-time update of the counter (if supported by your terminal) than the count itself? Generally we should work on streaming in test results where possible which is (sort of) tracked in #141, so I'll close this as a duplicate.

Do let me know if something was misunderstood here!

mds1 commented 2 years ago

Ah yea it does show the number of fuzz runs after a run already. The main motivation for opening this issue was to show a counter during the fuzz runs. I recently ran 10k fuzz runs with ffi, so it was quite slow, and it looks like your tests are hanging since there's nothing printed to the terminal during execution.

A real-time fuzz run counter seems to be a simpler scope than #141, and I don't see this functionality mentioned in there (though I only skimmed all the comments), so I feel like they could be tracked separately. But either way is fine with me 🙂

onbjerg commented 2 years ago

I agree it would be valuable to have some sort of real-time counter. Let's track it here then :sweat_smile: Can you update the original issue to reflect it?

mds1 commented 2 years ago

done!

lucas-manuel commented 2 years ago

I would love to see this for invariants

PaulRBerg commented 1 year ago

This would also be useful for long-winded end-to-end fork tests that also fuzz inputs, since they can take a long time to run.

brockelmore commented 6 months ago

We should add coverage % for invariant runs. Basically, record locations of JUMPDESTs in a contract, and when we hit it record that and print out something like:

Runs: 6097
Coverage: 54% (27/54 Blocks)