How can I do shorter benchmarks?

[chrissound]

module Main where

import Criterion.Types
import Criterion.Main

myConfig :: Config
myConfig = defaultConfig {
              resamples = 1
           , timeLimit = 0.01
           }

main :: IO ()
main = do
  let f = (\x -> bench (show x)  $ nf xyz x)
  defaultMainWith myConfig [
    bgroup "fib" [
                  env (pure 400000) f
                , env (pure 100) f
                , env (pure 200) f
                ]
    ]

xyz :: Int -> [Double]
xyz 0 = []
xyz _ = [fromIntegral 1000]

I was hoping I could quickly benchmark an example like this - but it seems to be taking around 4 seconds - to do numerous(?) runs.

Have I misunderstood the resamples / timeLimit values?

[RyanGlScott]

In order to ensure that criterion generates enough data for us to have some trust in the raw measurements, criterion sets a minimum time threshold that a benchmark must run for:

https://github.com/bos/criterion/blob/642bea6d15b9a1737c82ffd051f663edcb89d280/criterion-measurement/src/Criterion/Measurement.hs#L240-L250

This is where the threshold is used in the inner loop of benchmarking:

https://github.com/bos/criterion/blob/642bea6d15b9a1737c82ffd051f663edcb89d280/criterion-measurement/src/Criterion/Measurement.hs#L300-L315

If you do the math here, it will likely add up to the time you're seeing.

Admittedly, the Haddocks for timeLimit don't make mention of this fact, although they probably should.

[jberryman]

I have some benchmarks where a single iteration takes ~5min. Is there any alternative for me other than rewriting them outside of criterion? I essentially want to run them just once and have the time reported.