Sleeping in `Bencher::with_inputs` affects benchmark time

[OliverKillane]

Description

with_inputs affects the benchmark time with sleep, the documentation specifies it does not.

Replicate

Example here

[nvzqz]

Your benchmarks are measuring dropping Vec inputs because they use bench_values. Switching to bench_refs fixes the timings by deferring drop until after the loop.

Not sure why sleeping affects timings though.

[OliverKillane]

D'oh, forgot the drop. That fixes it for me - much appreciated!

Is the with_sleep worth pursuing, or should we close this issue?

[nvzqz]

I think sleeping affecting timing is a legit issue.

[OliverKillane]

I have fixed the linked example, with a simpler mm_pause for busy sleeping. I need to sleep now.

[nvzqz]

I see the issue here. You're setting max_time = 0.1 which also takes into account input generation time.

Internally it's hitting max_time because the benchmark loop needs more iterations but then with_inputs spends more of the time budget. This causes with_sleep to only have enough time to record a single sample of 1, 8, or 64 iterations. To get accurate timings it needs to record multiple samples over more iterations.

Setting DIVAN_SAMPLE_SIZE=100 (instead of dynamic sample size) shows that sleeping in input generation doesn't affect timings.