Clarification on changing the criterion iteration strategy?

[Boshen]

I'd like to try codspeed. The compact library runs faster because the criterion iteration strategy defers from the main library where it only runs a total of 6 times and only record the last one.

https://github.com/CodSpeedHQ/codspeed-rust/blob/a881b54f72020a0bcd73e64b32c155a2432d4b59/crates/criterion_compat/src/compat/bencher.rs#L23-L39

May I ask for the rational behind this decision?

[art049]

Hi, since we measure the performance with instrumentation(based off Valgrind), we don't need as much warmup as with "regular time-based measurements". Thus, those 5 first runs allow to warmup mainly the caches and allocators before measuring a single execution of the benchmark payload.

[Boshen]

Thanks!