dapphub / dapptools

Dapp, Seth, Hevm, and more
https://dapp.tools
2.1k stars 326 forks source link

Log gas for one property test run #692

Open transmissions11 opened 3 years ago

transmissions11 commented 3 years ago

Currently using property tests means you forfeit your ability to see logs or gas usage in the dapp test output, would like a way to still see debug information like that so using property tests is truly a no tradeoff feature.

This is a tough problem as a run may not fail but still may not represent the expected usage of the contract (ie: test function returns early because a number given as a parameter was too large). Could be solved with some sort of heuristics to detect a test run with a median gas usage (or some other metric)?

Could also be solved by giving the test developer a way to specify if a run functioned perfectly as expected (no early returns, etc).

wdyt?

d-xo commented 3 years ago

This would be nice. Maybe a median / mean with some standard deviation?

I can't really think of a simple way to allow users to specify what a "normal" execution looks like. I think if there's a particular path through a test function that's interesting to profile, maybe it makes more sense to just call the property test method with concrete inputs from a concrete test and check the gas usage there?

MrChico commented 3 years ago

Related to #349, this requires to use something handcrafted instead of quickcheck

d-xo commented 3 years ago

Ah nice. We need to drop quickcheck for the coverage guided fuzzer so this should get easier once that's merged.