Consider changing the default benchmark timer.

[hodgestar]

pytest-benchmark allows changing the default benchmark timer, for example, for the FAQ:

I can’t avoid using VMs or running other programs. What can I do? As a last ditch effort pytest-benchmark allows you to plugin in custom timers (--benchmark-timer). You could use something like time.process_time (Python 3.3+ only) as the timer. Process time doesn’t include sleeping or waiting for I/O.

-- see https://pytest-benchmark.readthedocs.io/en/latest/faq.html

Maybe process_time is not the best for some benchmarks, but it is an interesting option for removing variability on VMs, and perhaps it would even be possible to record both somehow.

[hodgestar]

It would be nice to run some tests on GitHub action runners to see how stable this makes the result (and thus whether it's worth the trade-off).

[xspronken]

Tests were made and process_time did not perform better than the default timer. In the images below the timer was changed from process_time to the default timer on 2022-08-31. We can not see any major changes in reliability and certain datatypes, notably the qutip ones perform faster.