If I have something like @b rand(1000) sort!, the first eval is much slower than subsequent evals within a given sample, which violates benchmarking assumptions and results in weird results. For example, @b rand(1000) sort! reports a super fast runtime while @b rand(100_000) sort! is realistic.
I guess most of these cases can be detected by systematically running a second evaluation after the first one? Of course it's debatable whether the benefit outweighs the cost
If I have something like
@b rand(1000) sort!
, the first eval is much slower than subsequent evals within a given sample, which violates benchmarking assumptions and results in weird results. For example,@b rand(1000) sort!
reports a super fast runtime while@b rand(100_000) sort!
is realistic.See: https://github.com/withbayes/Tapir.jl/issues/140