Using `with_inputs` or not

[cBournhonesque]

Hi,

I'm not sure I understand the use of with_inputs.

Let's say I have this benchmark:

#[divan::bench(
    types = [Ascii, Unicode],
    consts = LENS,
)]
fn to_ascii_uppercase<G: GenString, const N: usize>(bencher: Bencher) {
    let mut gen = G::default();
    bencher
        .counter(CharsCount::new(N))
        .with_inputs(|| gen.gen_string(N))
        .input_counter(BytesCount::of_str)
        .bench_local_refs(|s| s.to_ascii_uppercase());
}

I get that the time to create the strings wouldn't affect the benchmark.

But what about if I create the string inside bench_local, like this?

#[divan::bench(
    types = [Ascii, Unicode],
    consts = LENS,
)]
fn to_ascii_uppercase<G: GenString, const N: usize>(bencher: Bencher) {
    let mut gen = G::default();
    let s = gen.gen_string(N)
    bencher
        .bench_local(|| s.to_ascii_uppercase());
}

The time to create the string is also not included in the benchmark right? Therefore is the use of with_inputs mostly to have additional helpers such as input_counter?

[nvzqz]

You are correct that the first and second example do not include string generation time in benchmark timings. However, that's where the similarities end.

Differences:

• The first example generates a new string for every benchmark iteration. Whereas the second example reuses the same string instance. The first example improves over the second because it generalizes over various cases instead of a single case that may exhibit anywhere between best-case or worst-case performance repeatedly for every iteration. Also, the second case will easily enable the CPU's cache and branching systems to optimize for repeatedly working over the same data, which probably does not actually reflect real world work.
• The first example has Divan black_box every string, preventing the compiler's optimizer from making assumptions about the data. In the second example, Divan does not black_box the closure passed to bench/bench_local and thus the optimizer may allow the code to appear more optimal than it would in practice in real world scenarios.