PkgJogger

PkgJogger provides a framework for running suites of BenchmarkTools.jl benchmarks without the boilerplate.

Just write benchmarks

Create a benchmark/bench_*.jl file, define a BenchmarkTools.jl suite and go!

using BenchmarkTools
using AwesomePkg
suite = BenchmarkGroup()
suite["fast"] = @benchmarkable fast_code()

PkgJogger will wrap each benchmark/bench_*.jl in a module and bundle them into JogAwesomePkg

using AwesomePkg
using PkgJogger

# Creates the JogAwesomePkg module
@jog AwesomePkg

# Warmup, tune, and run all of AwesomePkg's benchmarks
JogAwesomePkg.benchmark()

Benchmark, Revise, and Benchmark Again!

PkgJogger uses Revise.jl to track changes to your benchmark/bench_*.jl files and reload your suite as you edit. No more waiting for benchmarks to precompile!

Tracked Changes:

• Changing your benchmarked function
• Changing benchmarking parameters (i.e. seconds or samples)
• Adding new benchmarks

Current Limitations:

• New benchmark files are not tracked
• Deleted benchmarks will stick around
• Renamed benchmarks will create a new benchmark and retain the old name

To get around the above, run @jog PkgName to get an updated jogger.

Note: Revise must be loaded first in order to track changes to your benchmark files.

Continuous Benchmarking Baked In!

Install PkgJogger, run benchmarks, and save results to a *.bson.gz with a one-line command.

julia -e 'using Pkg; Pkg.add("PkgJogger"); using PkgJogger; PkgJogger.ci()'

What gets done:

• Constructs a temporary benchmarking environment from Project.toml and benchmark/Project.toml.
• Creates a jogger to run the package's benchmarks.
• Warmup, tune and run all benchmarks.
• Save Benchmarking results and more to a compressed *.bson.gz file.

Or for a more lightweight option, use @test_bechmarks to run each benchmark once (No Warmup, tuning, etc.), as a smoke test against benchmarking regressions.