I've noticed that the numbers that criterion reports for this benchmark can vary quite a bit from run to run—sometimes a difference of about 20%. The culprit seems to be the list that gets randomly generated once and used in every iteration. I suspect the sorting algorithms perform much better depending on how well sorted the generated list is, which would account for the differing benchmark times.
My solution is to use the create function from mwc-random, which uses a fixed seed, instead of of withSystemRandom. This way, the same numbers will be generated for every run of the benchmarks.
snoyberg
commented
8 years ago
I've noticed that the numbers that
criterionreports for this benchmark can vary quite a bit from run to run—sometimes a difference of about 20%. The culprit seems to be the list that gets randomly generated once and used in every iteration. I suspect the sorting algorithms perform much better depending on how well sorted the generated list is, which would account for the differing benchmark times.My solution is to use the
createfunction frommwc-random, which uses a fixed seed, instead of ofwithSystemRandom. This way, the same numbers will be generated for every run of the benchmarks.