Open MangoIV opened 10 hours ago
Benchmarkable
doesn't really allow for this though atm.
funcToRandomBench
:: forall a b c.
(b -> c) -- ^ How to evaluate results, usually 'id' or 'rnf'.
-> (a -> b) -- ^ What to benchmark
-> IO a -- ^ Generator of inputs
-> Benchmarkable
funcToRandomBench frc = (Benchmarkable .) . funcToRandomBenchLoop SPEC
where
funcToRandomBenchLoop :: SPEC -> (a -> b) -> IO a -> Word64 -> IO ()
funcToRandomBenchLoop !_ f mx n
| n == 0 = pure ()
| otherwise = do
x <- mx
_ <- evaluate (frc (f x))
funcToRandomBenchLoop SPEC f mx (n - 1)
{-# INLINE funcToRandomBench #-}
Would this be enough? It's up to supplied IO a
whether to generate random inputs each time or pregenerate a list of them and iterate over it repeatedly.
You would probably still benchmark a single random input multiple times to get good confidence and then average over multiple random inputs. I tried to use perRunEnv with criterion with this random function and it gave me wild results. It might be similar here (though it might also be just fine?) I think the function you propose is pretty good for random inputs that are expected to have very similar runtimes.
You would probably still benchmark a single random input multiple times to get good confidence and then average over multiple random inputs.
If you limit IO a
to iterate over a finite list of random inputs, this would happen automatically in funcToRandomBench
, right?
I mean sure but I feel like this should ultimately be part of the interface, no?
Btw ~ really cool library, currently only using it to output stats in form of a csv but that’s much easier and quicker than with criterion ❤️
Hi! Say I have an algorithm that I expect to perform differently on a bunch of different inputs and say I have a sufficiently good
Gen
instance ~ why would it be a bad idea to generate a bunch of inputs withGen
and then average the benchmarks over these to get a good picture on how the algorithm behaves on different inputs?I feel like this would be kind of an obvious thing to do, not necessarily via
Gen
but maybe with an interface like this:and then maybe
What do you think?