Automatically benchmark PRs

[MilesCranmer]

This creates a GitHub action that uses AirspeedVelocity.jl to automatically run the benchmark/benchmarks.jl file on all submitted PRs, and create a table of the performance measurements. This is useful for catching performance regressions.

I tested it on your benchmarks and seems to work. Although I did have to create a Project.toml file in the benchmark file because Metatheory is imported in the benchmarks.

An example GitHub PR comment is shown here: https://github.com/SymbolicML/DynamicExpressions.jl/pull/27#issuecomment-1525571227

[codecov-commenter]

Codecov Report

Merging #522 (dc4b951) into master (ad5664a) will decrease coverage by 3.03%. The diff coverage is 79.26%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master     #522      +/-   ##
==========================================
- Coverage   83.37%   80.35%   -3.03%     
==========================================
  Files          10       14       +4     
  Lines        1257     1817     +560     
==========================================
+ Hits         1048     1460     +412     
- Misses        209      357     +148     

Impacted Files	Coverage Δ
src/substitute.jl	96.15% <ø> (ø)
src/utils.jl	58.92% <55.88%> (+1.18%)	:arrow_up:
src/rewriters.jl	56.19% <56.19%> (ø)
src/interface.jl	66.66% <66.66%> (ø)
src/methods.jl	81.25% <79.31%> (-0.15%)	:arrow_down:
src/types.jl	81.76% <81.22%> (-3.23%)	:arrow_down:
src/code.jl	73.18% <81.66%> (-0.69%)	:arrow_down:
src/rule.jl	83.70% <82.35%> (-12.46%)	:arrow_down:
src/simplify.jl	85.71% <85.71%> (ø)
src/polyform.jl	92.98% <86.84%> (+0.22%)	:arrow_up:
... and 3 more

... and 1 file with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[MilesCranmer]

Here's an example output I ran:

	v0.20.0	v1.0.5	t[v0.20.0]/t[v1.0.5]
overhead/acrule/a+2	0.951 ± 0.058 μs	0.887 ± 0.048 μs	1.07
overhead/acrule/a+2+b	0.926 ± 0.039 μs	0.859 ± 0.033 μs	1.08
overhead/acrule/a+b	0.363 ± 0.009 μs	0.336 ± 0.01 μs	1.08
overhead/acrule/noop:Int	0.039 ± 0.001 μs	0.037 ± 0.001 μs	1.05
overhead/acrule/noop:Sym	0.055 ± 0.003 μs	0.052 ± 0.005 μs	1.06
overhead/rule/noop:Int	0.054 ± 0.002 μs	0.055 ± 0.002 μs	0.982
overhead/rule/noop:Sym	0.072 ± 0.004 μs	0.06 ± 0.003 μs	1.2
overhead/rule/noop:Term	0.069 ± 0.003 μs	0.06 ± 0.003 μs	1.15
overhead/ruleset/noop:Int	0.138 ± 0.005 μs	0.14 ± 0.005 μs	0.986
overhead/ruleset/noop:Sym	0.169 ± 0.019 μs	0.161 ± 0.022 μs	1.05
overhead/ruleset/noop:Term	6.55 ± 0.32 μs	6.37 ± 0.22 μs	1.03
overhead/simplify/noop:Int	0.2 ± 0.004 μs	0.207 ± 0.006 μs	0.966
overhead/simplify/noop:Sym	0.233 ± 0.018 μs	0.239 ± 0.022 μs	0.975
overhead/simplify/noop:Term	0.0549 ± 0.0029 ms	0.0568 ± 0.0023 ms	0.965
overhead/simplify/randterm (+, *):serial	0.17 ± 0.0018 s	0.177 ± 0.0013 s	0.959
overhead/simplify/randterm (+, *):thread	0.182 ± 0.036 s	0.181 ± 0.036 s	1
overhead/simplify/randterm (/, *):serial	0.338 ± 0.013 ms	0.349 ± 0.0092 ms	0.967
overhead/simplify/randterm (/, *):thread	0.373 ± 0.01 ms	0.39 ± 0.011 ms	0.956
overhead/substitute/a	0.123 ± 0.0029 ms	0.126 ± 0.0028 ms	0.978
overhead/substitute/a,b	0.116 ± 0.0037 ms	0.119 ± 0.0022 ms	0.978
overhead/substitute/a,b,c	0.0791 ± 0.001 ms	0.0808 ± 0.0015 ms	0.979
polyform/easy_iszero	0.0538 ± 0.0011 ms	0.0552 ± 0.00074 ms	0.975
polyform/isone	25 ± 1 ns	19 ± 1 ns	1.32
polyform/iszero	1.94 ± 0.037 ms	1.98 ± 0.024 ms	0.98
polyform/simplify_fractions	2.97 ± 0.41 ms	3.08 ± 0.29 ms	0.964
time_to_load	1.3 ± 0.0078 s	1.31 ± 0.018 s	0.996

[shashi]

That is really cool! Thank you!