`@btime` reports wrong allocation

[chunjiw]

MWE:

using BenchmarkTools

struct S
    x
end
function (s::S)()
    s.x[1]
end

function f()
    s1 = S(rand(10))
    @time s1()
    @btime $s1()
    println(@allocated s1())
    s2 = S(1)
    @time s2()
    @btime $s2()
    @benchmark $s2()
    println(@allocated s2())
end

f()

The output I see is

  0.000001 seconds
  19.435 ns (1 allocation: 16 bytes)
0
  0.000000 seconds
  18.050 ns (0 allocations: 0 bytes)
0

So when the field is a vector, @btime reports wrong allocation, assuming @time and @allocated are right. This may be related to https://github.com/JuliaLang/julia/issues/53694 but I'm not sure.

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, broadwell)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)

[Zentrik]

It seems the @time reports no allocations as the compiler can see that s1 is constructed and then the first field is accessed and so can figure out that the type of s1() is Float64. Whilst for BenchmarkTools the s1() call happens in a non-inlined function so it cannot do the same optimisation.

It's not obvious that it could be worked around, nor that we would want to.