Bodigrim / tasty-bench

Featherlight benchmark framework, drop-in replacement for criterion and gauge.
https://hackage.haskell.org/package/tasty-bench
MIT License
80 stars 11 forks source link

Benchmarks sometimes get stuck #46

Closed konsumlamm closed 1 year ago

konsumlamm commented 1 year ago

I unfortunately don't have a reproducible example, as this doesn't happen deterministically, but sometimes the benchmarks seem to get stuck. Restarting the benchmarks usually fixes the problem.

Bodigrim commented 1 year ago

And no suggestions to set -t100in stderr? You might try setting debug build flag.

konsumlamm commented 1 year ago

Eventually, I get

This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning).

Is this to be expected? The benchmarks shouldn't vary too much and when rerunning, it doesn't occur, so this feels like some bug.

Bodigrim commented 1 year ago

This is expected, either set --timeout 100, or alternatively increase --stdev.