bheisler / criterion.rs

Statistics-driven benchmarking library for Rust
Apache License 2.0
4.31k stars 292 forks source link

`Criterion::bench_with_input` is easy to misuse #607

Open MinusKelvin opened 1 year ago

MinusKelvin commented 1 year ago

The section for Benchmarking With One Input in the Criterion User Guide says:

This is convenient in that it automatically passes the input through a black_box so that you don't need to call that directly.

However, this is not sufficient to prevent the compiler from optimizing the benchmark away. It produces code analogous to:

let p = black_box(param);
loop {
    black_box(function(p));
}

After function is inlined (and possibly without inlining via other kinds of analysis), the compiler may perform loop invariant code motion, transforming it into code analogous to:

let p = black_box(param);
let v = function(p);
loop {
    black_box(v);
}

This can result in claimed running times on the order of 1 CPU cycle.

ryanseipp commented 1 year ago

I hit this issue today. Was benchmarking small functions that were defined in the same file the benchmarks were. I was getting results of 0.00 ps per benchmark run. The functions had #[inline] annotations, so I removed those. For one function, it was so small I needed to add #[inline(never)] to prevent it from being inlined.