search

NeuroBench / neurobench

Benchmark harness and baseline results for the NeuroBench algorithm track.
https://neurobench.readthedocs.io
Apache License 2.0
55 stars 12 forks source link

Add 'quiet' Parameter to Improve Benchmark Output Handling #144

Closed ben9809 closed 11 months ago

ben9809 commented 11 months ago

This pull request introduces a 'quiet' parameter to the benchmark.run() function, addressing the issue of excessive output generated during quick benchmark runs.

Related Issue: https://github.com/NeuroBench/neurobench/issues/142#issue-1957995144

ben9809 commented 11 months ago

Using redirect_stdout ensures that all output is consistently redirected (in our case suppressed), including output generated by any external libraries or functions called within the benchmark. It is a more elegant and effective means of controlling the benchmark's output behavior based on the quiet parameter.

jasonlyik commented 11 months ago

Looks nice Benedetto.

This is slightly related thought - what do you think about a "verbose" option which outputs/logs the benchmark results at every evaluation batch? Thus we can get faster feedback per-batch without needing to wait for the whole workload to finish.

ben9809 commented 11 months ago

Absolutely, it's a fantastic idea to include a "verbose" option.

Do you prefer a logging mechanism for a more structured approach, or do you prefer to use the print function for rapid feedback?

jasonlyik commented 11 months ago

This could be in-line with #32. For now we only have print statements and I think this would be most simple, but in the future we can have a clean logger.