Feature request: asynchronously build a benchmark group

[filchristou]

Please see first posts in https://discourse.julialang.org/t/how-to-asychronously-build-a-benchmark-group/106335

Some benchmarks have certain interdependencies and it would be nice to yield benchmark results while going through some mutating operations.

E.g. instead of doing

x = init()
SUITE["muttfunc1"] = @benchmarkable muttfunc1!(y) setup=(y = x |> deepcopy)
SUITE["muttfunc2"] = @benchmarkable muttfunc2!(y) setup=(y = x |> deepcopy |> mutfunc1!)
SUITE["muttfunc3"] = @benchmarkable muttfunc3!(y) setup=(y = x |> deepcopy |> mutfunc1!  |> mutfunc2!)

do something like

x = init()
@startbenchmark SUITE setup=(y=deepcopy(x))
  @yieldbenchmeasure "muttfunc1" muttfunc1!(y)
  @yieldbenchmeasure "muttfunc2" muttfunc2!(y)
  @yieldbenchmeasure "muttfunc3" muttfunc3!(y)
end

where a single sample will evaluate muttfunc1! , muttfunc2!, muttfunc3! once and will yield the result to the respecting BenchmarkGroup. So evals=1 needs to be hardcoded.

An also appealing interface would be to build such benchmarking functions e.g.

@benchmarkablegroup function mybenchfunc(x)
  @yieldbenchmeasure "muttfunc1" muttfunc1!(x)
  @yieldbenchmeasure "muttfunc2" muttfunc2!(x)
  @yieldbenchmeasure "muttfunc3" muttfunc3!(x)
end

The idea is similar to https://github.com/KristofferC/TimerOutputs.jl, but with a focus on BenchmarkingTools.jl

First please let me know if you would like this feature here If yes, which interface do you prefer ? If that's clear, I (or someone else) might commit to get this going.

[vchuravy]

BenchmarkTools is intentionally narrow in the problem it tries to solve. I think what you want is out of scope for BenchmarkTools

[filchristou]

Thanks for the fast answer. I guess then an external package would be the way to go. :)