Second function in `compare` reproducibly benchmarks faster than the first

SiriusStarr commented 4 years ago

With the use of compare, if identical functions are supplied, the second will reproducibly benchmark ~10-30% faster than the first, regardless of what the functions are.

SSCCE:

module Test exposing (main)

import Benchmark exposing (Benchmark)
import Benchmark.Runner exposing (BenchmarkProgram, program)

main : BenchmarkProgram
main =
    program suite

suite : Benchmark
suite =
    Benchmark.compare "First vs. Second" "First" (\_ -> myFunction "zxcvbn") "Second" (\_ -> myFunction "zxcvbn")

myFunction : String -> String
myFunction =
    String.reverse

and some results for various functions in myFunction:

This behavior extends to more complex functions as well, just presenting simple ones here so as to eliminate variables.

gampleman commented 3 years ago

Hm I'm struggling to reproduce this.

I see much smaller variance (the largest I observed was -2.4%) on both Chrome and Safari. The second was usually the slower one, but occasionally the faster one.

SiriusStarr commented 3 years ago

(Today on NixOS, over about 20 runs each, with String.reverse)

In Firefox 90.0.2, currently I'm getting about ~3-4% faster for function 2 on average, with the largest I've seen being about 6.5%.
In Chromium 92.0.4515.107, I reliably see about 9-11% faster for function 2.

Tried using elm 0.19.0 and also v1.0.1 of benchmark and behavior is similar, so it doesn't seem to have been altered by either of those updates. I might have been on Fedora rather than NixOS at the time, but don't remember as it was a year and a half ago.

This is apparently very specific to the individual's computer, so may be challenging to reproduce/track down. :disappointed:

elm-explorations / benchmark

Second function in `compare` reproducibly benchmarks faster than the first #18