Add experimental support for customisable benchmarking

[Zentrik]

Replaces #325 (closes #325)

This pr adds the ability to run a custom benchmarking function which has hooks to inject in custom functions. The current design supports running perf on a benchmark (see https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl) and profiling benchmarks (excluding the setup, teardown and gcscrub which the current bprofile includes).

[Zentrik]

Do we need to still support Julia 1.0 and 1.5?

[Zentrik]

All tests should be passing now except for the type piracy tests. Do these methods need to be moved into LinuxPerf, how do I avoid type piracy? https://github.com/JuliaCI/BenchmarkTools.jl/blob/b3f66b2608088115c028dda0355171e549da707b/src/serialization.jl#L35-L50

EDIT: Created pr to move into LinuxPerf.jl, https://github.com/JuliaPerf/LinuxPerf.jl/pull/35.

[Zentrik]

Also right now it seems we run perf every sample but only store the last sample's result. Do we want to change this. If so do we want to store them all or should we only run perf on the last sample.

EDIT: A bit tricky to only run perf once as we need access to setup and teardown expressions in _run. I don't think this should be blocking.

EDIT 2: perf now only runs on the last sample.

[Zentrik]

I haven't added tests, but given the CI doesn't have perf available, not sure how useful it would be.

[DilumAluthge]

I haven't added tests, but given the CI doesn't have perf available, not sure how useful it would be.

We might need to set up Buildkite CI on this repo. @vchuravy @staticfloat

[Zentrik]

I think main things left to do are:

• [x] Passing custom options to Linux Perf doesn't work currently - I'm going to leave this for a follow up pr as I think this is going to require a fair amount of work to do well and doesn't strike me as particularly important. Let me know if there are any objections to that.
• [x] Add CI that can run with perf and tests
• [x] See how breaking this is - I've added constructors for Trial and TrialEstimate that should mitigate any breakages.
• [x] Add tests

[willow-ahrens]

I'm not sure I have the bandwidth as a maintainer of this package to support this kind of infrastructure directly in benchmarktools.jl. Is there a way that we can support this inside LinuxPerf.jl rather than in BenchmarkTools.jl? Maybe a separate run_perf function that runs the benchmarks in perf?

[willow-ahrens]

it seems like too much overhead to me to support separate CI for this just to support this feature here, especially if it's possible to support this in linuxperf.jl. Could linuxperf support a separate run_perf function that runs a BenchmarkTools Benchmark through perf?

Especially if we need special CI to run some software, it seems much easier to support this kind of feature in that repo, rather than this one.

[Zentrik]

I think if I can get this to a low enough maintenance overhead for you it would be best to add perf profiling in BenchmarkTools. I think we can just remove the extra CI and make it clear that this is experimental. I don't think there's much need to make sure the profiling works on CI and users can easily run their own tests. Let me know if there's anything else I can do to lower the overhead or if it's not going to be possible for the overhead to be low enough.

If you care why I don't think it makes sense to put this functionality in LinuxPerf or BaseBenchmarks, I've explained below. So running perf on samplefunc seems to work ok, it has fairly high overhead but it seems to be not much noisier than this pr which is what really matters. However, actually integrating this in LinuxPerf or BaseBenchmarks seems difficult as presumably they would have to overload BenchmarkTools._run so that they can run perf whilst the function is still hot when benchmarking a BenchmarkGroup. I don't think we would want to do this by default for people using LinuxPerf so if I was to do this I would probably put it in BaseBenchmarks which doesn't have CI anyways. Also, they would have to setup a way to save and load the perf profiling results.

[willow-ahrens]

Could we add something like a prehook and a posthook argument to the run function, so that other packages can export their own versions of run with special behavior, such as bprofilerun, bprofileviewrun, bperfrun? Is there a way to make them nest?

[willow-ahrens]

I'm imagining something like:

run_perf(args...; kwargs..., posthook=f) = run(args...; kwargs..., posthook=(b)->(run_perf_stuff(b); f(b)))

[Zentrik]

I'd like to mark this feature as experimental or something to that effect so that we can make breaking changes to it without making a breaking change to BenchmarkTools. The changes to samplefunc are largely just so that it matches customisable_func (theoretically it reduces overhead by a couple of instructions but that's not noticeable) and also so the hooks for samplefunc are exposed.

I'll make a separate pr removing Buildkite.

I've moved the LinuxPerf stuff to https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl.

I do have a version of bprofile that uses this functionality to only profile the relevant stuff, but because samples are generally quite short (on the order of microseconds) not much gets profiled making it pretty useless.

bprofile_setup_prehook(_) = Profile.check_init()
function bprofile_prehook()
    results = samplefunc_prehook()
    status = ccall(:jl_profile_start_timer, Cint, ())
    if status < 0
        error(Profile.error_codes[status])
    end
    return results
end
function bprofile_posthook()
    Profile.stop_timer()
    return samplefunc_posthook()
end

# """
#     @bprofile expression [other parameters...]

# Run `@benchmark` while profiling. This is similar to

#     @profile @benchmark expression [other parameters...]

# but the profiling is applied only to the main
# execution (after compilation and tuning).
# The profile buffer is cleared prior to execution.

# View the profile results with `Profile.print(...)`.
# See the profiling section of the Julia manual for more
# information.
# """
macro bprofile(args...)
    _, params = prunekwargs(args...)
    if !haskw(args, :gctrial)
        args = (args..., Expr(:kw, :gctrial, false))
    end
    if !haskw(args, :gcsample)
        args = (args..., Expr(:kw, :gcsample, false))
    end
    tmp = gensym()
    return esc(
        quote
            local $tmp = $BenchmarkTools.@benchmarkable $(args...)
            $(
                if hasevals(params)
                    :(run(
                        $tmp, $BenchmarkTools.Parameters($tmp.params; evals=1); warmup=false
                    ))
                else
                    :($BenchmarkTools.tune!($tmp))
                end
            )
            $BenchmarkTools.Profile.stop_timer()
            $BenchmarkTools.Profile.clear()
            $BenchmarkTools.run(
                $tmp;
                setup_prehook=$BenchmarkTools.bprofile_setup_prehook,
                prehook=$BenchmarkTools.bprofile_prehook,
                posthook=$BenchmarkTools.bprofile_posthook,
                sample_result=$BenchmarkTools.samplefunc_sample_result,
                enable_customisable_func=:ALL,
                run_customisable_func_only=true,
            )
        end,
    )
end