LilithHafner
commented
6 months ago You're benchmarking the runtime of a noisy function that takes >100ms with the default benchmarking runtime budget of 100ms. Consequently, all Charimarks can do while respecting your runtime budget is a single noisy evaluation. When you see strange results, it can often be helpful to run @be instead of @b to see what's going on. On a quiet system with low memory pressure, you may get results like this:
x@fedora:~$ julia
o | Version 1.11.0-alpha1 (2024-03-01)
o o | Official https://julialang.org/ release
julia> using Chairmarks
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
88.744 ms (3 allocs: 762.940 MiB, 4.49% gc time)
90.265 ms (3 allocs: 762.940 MiB, 6.25% gc time)
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
89.377 ms (3 allocs: 762.940 MiB, 4.55% gc time)
89.461 ms (3 allocs: 762.940 MiB, 5.54% gc time)
julia> using BenchmarkTools
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
91.414 ms (3 allocs: 762.940 MiB, 4.48% gc time)
92.910 ms (3 allocs: 762.940 MiB, 6.91% gc time)
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
90.399 ms (3 allocs: 762.940 MiB, 4.45% gc time)
90.649 ms (3 allocs: 762.940 MiB, 4.60% gc time)
julia> @btime rand(100, 10000, 100);
89.504 ms (3 allocations: 762.94 MiB)
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
91.491 ms (3 allocs: 762.940 MiB, 4.79% gc time)
95.220 ms (3 allocs: 762.940 MiB, 5.61% gc time)
julia> @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
91.431 ms (3 allocs: 762.940 MiB, 4.82% gc time)
91.563 ms (3 allocs: 762.940 MiB, 4.73% gc time)But if you see an unacceptable level of variability for your usecase and only a single sample, you should increase the runtime budget.
A manual way for gc configuration may help?
The current syntax for triggering garbage collection before benchmarking is GC.gc(); @b ..., but it doesn't seem to change anything in your case (at least not on my machine). If anything, it adds a wee bit more noise.
julia> using Chairmarks
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
90.845 ms (2 allocs: 762.940 MiB, 5.37% gc time)
98.337 ms (2 allocs: 762.940 MiB, 10.28% gc time)
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
90.597 ms (2 allocs: 762.940 MiB, 4.96% gc time)
91.979 ms (2 allocs: 762.940 MiB, 5.56% gc time)
julia> using BenchmarkTools
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
91.146 ms (2 allocs: 762.940 MiB, 5.64% gc time)
98.766 ms (2 allocs: 762.940 MiB, 10.12% gc time)
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
91.029 ms (2 allocs: 762.940 MiB, 5.59% gc time)
91.210 ms (2 allocs: 762.940 MiB, 4.92% gc time)
julia> @btime rand(100, 10000, 100);
86.141 ms (2 allocations: 762.94 MiB)
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
86.430 ms (2 allocs: 762.940 MiB)
91.915 ms (2 allocs: 762.940 MiB, 5.50% gc time)
julia> GC.gc(); @be rand(100, 10000, 100)
Benchmark: 2 samples with 1 evaluation
92.999 ms (2 allocs: 762.940 MiB, 5.36% gc time)
101.280 ms (2 allocs: 762.940 MiB, 10.08% gc time)Do you see an improvement when running GC.gc()?
XLin0mu
Chairmarks.jl can't get the best performance before excuting the methods from BenchmarkTools. See my test:
Most strangly, It seems like happenning randomly. Sometime this trick won't work, too. Also, the performance fluctuating randomly even not doing anything, sometime can reach 200% times comparing with base time given by best-performanced
@band@btime. A manual way for gc configuration may help?