Hot code performs differently from cold code. Caches and stuff.
Currently execution order goes like this: benchmark0/sample0, benchmark0/sample1, ..., benchmark0/sampleN, benchmark1/sample0, benchmark1/sample1, ..., benchmark1/sampleN, ..., benchmarkM/sample0, benchmarkM/sample1, ..., benchmarkM/sampleN. This means that all executions of a single benchmark are sequenced together, which can lead to differences from caching effects.
It should be possible to execute different orders, like: benchmark0/sample0, benchmark1/sample0, ..., benchmarkM/sample0, benchmark0/sample1, benchmark1/sample1, ..., benchmarkN/sample1, ..., benchmark0/sampleN, benchmark1/sampleN, ..., benchmarkM/sampleN. This would alternate sampling between different benchmarks and reduce cache effects of locality.
Hot code performs differently from cold code. Caches and stuff.
Currently execution order goes like this: benchmark0/sample0, benchmark0/sample1, ..., benchmark0/sampleN, benchmark1/sample0, benchmark1/sample1, ..., benchmark1/sampleN, ..., benchmarkM/sample0, benchmarkM/sample1, ..., benchmarkM/sampleN. This means that all executions of a single benchmark are sequenced together, which can lead to differences from caching effects.
It should be possible to execute different orders, like: benchmark0/sample0, benchmark1/sample0, ..., benchmarkM/sample0, benchmark0/sample1, benchmark1/sample1, ..., benchmarkN/sample1, ..., benchmark0/sampleN, benchmark1/sampleN, ..., benchmarkM/sampleN. This would alternate sampling between different benchmarks and reduce cache effects of locality.
Random ordering is also a possibility.