search

rbenchmark / benchmarks

Collections of Benchmarks of R
BSD 3-Clause "New" or "Revised" License
10 stars 7 forks source link

Fix negative metrics #10

Closed llbit closed 8 years ago

llbit commented 8 years ago

Benchmarks could generate negative metrics because the warmup running time was subtracted from the main benchmark running time.

The running time for N=warmup_rep warmup runs is not the same for each benchmark invocation, and in some cases running N warmup runs takes longer time than the subsequent run of N warmup runs plus M=bench_rep benchmark runs. This is more common for shorter-running benchmarks. Negative statistics are also more frequent with the option --meter perf.

This commit makes the benchmark suite only measure the running time of M benchmark runs, and excludes the N warmup runs from the main benchmark run.

fixes #5

llbit commented 8 years ago

This also fixes the error that caused perf results to be discarded.

llbit commented 8 years ago

The problem with negative metrics only occurs for short-running benchmarks. The metrics errors are negligible for longer running benchmarks, so this fix is probably not useful for the longer-running benchmarks. Especially since omitting the warmup runs does not work well when benchmarking JITing RVM implementations.

I think a better solution is to adjust the number of warmup runs and benchmark runs to narrow the error margins for short-running benchmarks.