Add a switch to measure wall clock time

Bodigrim / tasty-bench

Featherlight benchmark framework, drop-in replacement for criterion and gauge.

https://hackage.haskell.org/package/tasty-bench

MIT License

80 stars 11 forks source link

Add a switch to measure wall clock time #23

Closed Bodigrim closed 2 years ago

amesgen commented 2 years ago

I would be interested in adding a way to do this:

At which level should the flag exist? Cabal flag or a new field in Benchmarkable or sth else?
How to measure (monotonic) time? getMonotonicTimeNSec (only available since GHC 8.4), a new external library like clock or criterion-measurement, or custom FFI?

Bodigrim commented 2 years ago

This should be customizable on per-benchmark level, something akin to

data Mode = CpuTime | WallTime 
instance Options Mode where ...

For time measurement I'd use https://hackage.haskell.org/package/tasty-1.4.2.1/docs/Test-Tasty-Runners.html#v:getTime. Alternatively getMonotonicTimeNSec is probably also fine.

What's your motivation to work on this? Do you have a use case, where measuring CPU time is not sufficient?

amesgen commented 2 years ago

So make Mode a tasty option via IsOption? And thanks for the pointer to getTime in tasty, completely forgot that tasty also has to measure monotonic time.

Do you have a use case, where measuring CPU time is not sufficient?

Yes, I played around with sth where I have to call out to subprocesses (in my case, z3), and I want that to be part of the benchmarked time.

Bodigrim commented 2 years ago

Yes, instance IsOption Mode. This allows to specify mode as a command line argument, but more importantly supports also fine-grained tuning via setOption: you likely want to measure wall time only for benchmarks of subprocesses, but not for anything else.