iu-parfunc / HSBencher

General benchmarking framework. Especially good at parameter studies.
13 stars 8 forks source link

Support linear regression in a general way #46

Open rrnewton opened 10 years ago

rrnewton commented 10 years ago

As of Criterion 1.0, it uses a linear regression methodology to generate an estimate of the expected marginal cost of running a benchmark just one more time.

A core type in Criterion is the Benchmarkable data structure, which has an exposed constructor and is of the form:

data Benchmarkable = Benchmarkable (Int64 -> IO ()) 

Usually, the user doesn't construct one of these objects directly, rather criterion constructs the Int64 -> IO () function that simply takes a number and runs an IO action N times. However, there are useful applications of constructing this function directly:

The reason I'm mentioning this here, is that HSBencher/Criterion integration would make sense, and could take two distinct forms:

In the second case, it's not clear to me whether the benchmarks should run in the same process as the benchmark harness or not. Opinions on that welcome.

rrnewton commented 10 years ago

CC @tmcdonell @svenssonjoel . Btw, one reason we want something like this is projects like ConcurrentCilk have some benchmarks where we want to do regression, but they don't use Haskell at all. (Except in the HSbencher script.)