Open rrnewton opened 10 years ago
CC @tmcdonell @svenssonjoel . Btw, one reason we want something like this is projects like ConcurrentCilk have some benchmarks where we want to do regression, but they don't use Haskell at all. (Except in the HSbencher script.)
As of Criterion 1.0, it uses a linear regression methodology to generate an estimate of the expected marginal cost of running a benchmark just one more time.
A core type in Criterion is the
Benchmarkable
data structure, which has an exposed constructor and is of the form:Usually, the user doesn't construct one of these objects directly, rather criterion constructs the
Int64 -> IO ()
function that simply takes a number and runs an IO actionN
times. However, there are useful applications of constructing this function directly:\n -> do init; realStuff n
.Par
monad. Doing arunPar
sets up an execution environment. This is an example of a one-time cost that can be amortized over a loop that runs /inside/ the Par monad.The reason I'm mentioning this here, is that HSBencher/Criterion integration would make sense, and could take two distinct forms:
In the second case, it's not clear to me whether the benchmarks should run in the same process as the benchmark harness or not. Opinions on that welcome.