a warning for sub-ns timings

[tpapp]

Results like

julia> using BenchmarkTools

julia> foo(x) = exp(x)
foo (generic function with 1 method)

julia> @btime foo(1)
  0.029 ns (0 allocations: 0 bytes)
2.718281828459045

confuse many discussions on the discourse forum and have limited real-world applicability.

Perhaps for timings below some threshold (eg 1ns, or something calculated from the CPU frequency, as much as that is meaningful) a warning could be printed indicating that the result is not necessarily meaningful in a nontrivial context.

[maxbennedich]

A timing this low seems to stem from when the expression is replaced with a constant:

julia> @btime 1
  0.029 ns (0 allocations: 0 bytes)

I can see how it'd be useful to detect this and warn about it. But, with that said, the more common problem I see on discourse is not sub-ns timings, but timings in the low nanoseconds; examples here, here, here, here, or this:

julia> @btime (s = 0.0; for n=1:1000 s += sqrt(n) end)
  1.894 ns (0 allocations: 0 bytes)

Even much larger timings, caused by the same phenomenon (compiler optimization), causes confusion, as in this topic. So in general, I think it's a hard problem to solve. Warning for sub-ns timings but not for low-ms timings might give the impression that the latter one is valid even when it's not.

What might help a bit is if this was more prominently mentioned in the documentation, perhaps a mention of it under "Quick Start", and a more thorough discussion under "Understanding compiler optimizations", including tips on how to work around it. If nothing else, we'd have something to link to in discourse posts.

[tpapp]

A tutorial on benchmarking would always be nice, but this issue is primarily meant to address one commonly occurring problem with a simple warning. I guess that a low-ns warning may be appropriate, too.

[jrevels]

something calculated from the CPU frequency

I think this would be the right approach. The message can be stated in a way that allows users to easily interpret the warning and decide for themselves what to do next. Something like:

"WARNING: Predicted minimum runtime is less than the time required for a single CPU cycle, based on the max CPU frequency reported by your OS (X Ghz). This often means that the Julia compiler has completely optimized away your benchmark kernel. If you believe this is happening, try using the setup keyword argument to feed dynamically valued inputs to your kernel (e.g. instead of @btime sin(1), try @btime sin(x) setup=(x=rand()))."