Console logs should ideally print _during_ fuzzing, not just after testing is complete

[webthethird]

Component

Forge

Describe the feature you would like

It would be ideal if the console logs would print during the fuzzing, rather than only after the testing is complete. Fuzzing is often a long process, and I'd like some insight into what it is doing while it is running. Especially since nothing else is displayed while fuzzing, not even the current number of runs.

Additional context

No response

[webthethird]

I realize a workaround for this is to write my logs to a .log file using vm.writeLine, but ideally file system permissions shouldn't be necessary to get real-time insight into a fuzzing campaign as it runs

[mds1]

@webthethird wdyt of closing this as a duplicate of https://github.com/foundry-rs/foundry/issues/585? It sounds like you just want confirmation things are still running and that the fuzzer isn't hanging. Or do you actually want to see the values being used during the fuzzing? If so I'd be curious to hear more about why

[webthethird]

@mds1 I'd actually like to see the values being used during fuzzing, since I'd like to make sure my tests don't have any bugs that would cause fuzzing to either succeed when they should be failing or take longer than they should.

For example, by implementing a workaround of writing the logs to a .log file using vm.writeLine I was able to a) realize mid-fuzz that I was using vm.store on the wrong slot and b) discover a potential bug in Forge (see issue #3848). In the first instance I was able to quickly stop fuzzing, fix my typo and re-run the fuzzer to get the results I was expecting, and the merits of the second point need no explanation!

[Azleal]

hi @grandizzy please give some advice on this topic. I'd like to give it another try if the approach is fully discussed. also, I have my own idea, how about setup a flag with which user could decide whether console.log in fuzz test should be printed or not. and the default value could be set as false, thus will not affect any fuzz tests prior to the change.

[grandizzy]

hi @grandizzy please give some advice on this topic. I'd like to give it another try if the approach is fully discussed. also, I have my own idea, how about setup a flag with which user could decide whether console.log in fuzz test should be printed or not. and the default value could be set as false, thus will not affect any fuzz tests prior to the change.

hey @Azleal , yeah, I think a config for fuzz / invariant test printing console logs as they're executed makes sense (cc @mds1 ) Also would be neat to print them not only at the end, but as they go (as requested by this issue), that is after each call, e.g. for invariants

https://github.com/foundry-rs/foundry/blob/8dd7ad2b22a8d7aca13f2e9d73c121a41da28baf/crates/evm/evm/src/executors/invariant/mod.rs#L315-L329

and for regular fuzz tests

https://github.com/foundry-rs/foundry/blob/8dd7ad2b22a8d7aca13f2e9d73c121a41da28baf/crates/evm/evm/src/executors/fuzz/mod.rs#L210-L213

Side effect would be that if running with both show logs flag and show progress then progress will become flaky, but I guess that's acceptable.

[grandizzy]

@Azleal lmk if you already looking into this, otherwise I'll pick it. thank you!

[Azleal]

@Azleal lmk if you already looking into this, otherwise I'll pick it. thank you!

hi @grandizzy sorry for my delayed reply. yes, I've already into this. I have tried to solve this issue once, so I'd like to give it another try. but i'm new to adding a new config, would appreciate it if you could give some suggestions or code referrences about adding and using a config in foundry. thanks

[grandizzy]

@Azleal no worries, thanks for looking into! you would need to add a bool like show_execution_logs in FuzzConfig (default false) here https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/config/src/fuzz.rs#L10-L12 Additionally you can make it inline / per test configurable by merging it here (I think this would be nice) https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/config/src/fuzz.rs#L68-L92

You will need to update test Configs here https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/forge/tests/cli/config.rs#L29-L30 and here https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/forge/tests/it/test_helpers.rs#L80 and I think that should be it

[Azleal]

@Azleal no worries, thanks for looking into! you would need to add a bool like show_execution_logs in FuzzConfig (default false) here

https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/config/src/fuzz.rs#L10-L12

Additionally you can make it inline / per test configurable by merging it here (I think this would be nice) https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/config/src/fuzz.rs#L68-L92

You will need to update test Configs here

https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/forge/tests/cli/config.rs#L29-L30

and here https://github.com/foundry-rs/foundry/blob/034393cc25fe84e35d89f5066775d9088db1de57/crates/forge/tests/it/test_helpers.rs#L80

and I think that should be it

thank you @grandizzy , lemme check it and I'll let you know if there's any updates.

[grandizzy]

going to close this one as an impl was added in #8387 This is a little bit different than printing during fuzzing - logs are printed when test ends to make sure they're not mixed (to view test progress --show-progress option can be used).