Open asinghvi17 opened 7 months ago
What? I had not idea. This is lovely :)
Is there a way to take a BenchmarkGroup and "translate" it somehow? Or otherwise run a benchmark suite using Chairmarks instead of BenchmarkTools? Many projects have historically defined their benchmark suite in benchmarks/benchmarks.jl
and use BenchmarkTools. So to measure performance over time it is perhaps not practical to replace all previous @benchmarkable
(especially if there are setup=...
in the macro).
I ask specifically in the context of AirspeedVelocity.jl: https://github.com/MilesCranmer/AirspeedVelocity.jl/issues/35
I mean they both have the same syntax, so an operation at all leaf nodes of that dict could easily reassign the @benchmarkable
s to the results of @be
on that same Expr
(if you can directly invoke a macro on an Expr).
I think they have slightly different syntax, no?
Positional argument disambiguation
≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡
setup, teardown, and init are optional and are parsed with that precedence giving these possible forms:
@be f
@be setup f
@be setup f teardown
@be init setup f teardown
whereas @benchmarkable
is
help?> @benchmarkable
@benchmarkable <expr to benchmark> [setup=<setup expr>]
Create a Benchmark instance for the given expression. @benchmarkable has similar syntax with @benchmark. See also @benchmark.
Ah that's true - I had not used setup
much but I guess it could also be translated by way of Expr rewriting
Wait, I'm a bit confused. @benchmarkable
returns a benchmark (that you can then execute). Whereas @be
appears to actually return the results of benchmarking the expression. Is that correct?
Yep! Creating a benchmark suite with @be
instead of @benchmarkable
gives you the equivalent of that suite after run
.
Oh, but isn't the whole point of @benchmarkable
for it to be lazily evaluated, so you can tune!
it? And @benchmark
for eager evaluation?
That's true, but if running in a non-interactive framework I don't think it really matters?
Even then, the performance difference is substantial enough that it's actually possible to do semi-interactive workflows with Chairmarks.
I guess I just wouldn't consider @benchmarkable
-> @be
to be a complete solution (maybe this deserves a new issue with a feature request). For example if I need a single benchmark result I would use @benchmark
. But I want a suite of benchmarks that I can store in my REPL, in between Revise.jl-ing my library, then the suite
is something I would want to re-run. In principle it doesn't seem too bad to add a compatibility layer?
So perhaps a ext/ChairmarksBenchmarkToolsExt.jl
could create a @benchmarkable
that still uses benchmarkable_parts
to extract the pieces, but uses Chairmarks instead for running the thing? I'm not sure how doable this is. Maybe @LilithHafner could share their thoughts.
Hmm! Yeah that logic could probably be directly translated to Chairmarks somehow, probably by creating a new macro which stores an Expr
object when run (the equivalent of @benchmarkable
). It doesn't look like this is hijackable unless either we do major type piracy by overloading BenchmarkTools.generate_benchmark_definition
directly, or offer a function Chairmarks.override_benchmarkable!()
to @eval
that code in.
Simply replacing
@benchmarkable
with@be
suffices, and you don't have to runtune!
orrun
either!Even running
Statistics.median(suite)
works - although any custom plotting utilities might need a couple of tweaks :)