Benchmarks for Functor Overhead

[bosonbaas]

We are interested in obtaining and documenting benchmark results to determine the overhead of using functors with GAT expressions versus just directly using the Julia libraries referenced in instances. This problem is related to issue #125, inasmuch as we would like to show that using the "backend agnostic representations of linear operators" is just as efficient as directly using LinearOperators or LinearMaps (has trivial overhead). The relevant questions are: Is there interest in this benchmarking being implemented generally (for more than just instances of GLA)? What are some representative functions that would be useful to benchmark for speed and memory?

[bosonbaas]

From my research, BenchmarkTools.jl provides an easy-to-use benchmarking API which would allow us to collect functions run with varying parameters (matrix size, sparsity, etc.) to highlight the impact of our overhead. This would also allow us to easily integrate with other libraries like PkgBenchmark.jl and BenchmarkCI.jl if we wish to eventually automate these benchmarks and add them to Github Actions.

[1] https://github.com/JuliaCI/BenchmarkTools.jl [2] https://github.com/JuliaCI/PkgBenchmark.jl [3] https://github.com/tkf/BenchmarkCI.jl

[jpfairbanks]

Yeah BenchmarkTools is definitely the way to go. @epatters how do you want to handle dependencies for benchmarking stuff? Should we add a folder to this repo like /test/benchmarks and then include BenchmarkTools as a test time dependency? Or should we set up another repo for CatlabBenchmarks.jl and do all the benchmarking in there?

[epatters]

I am fine with either approach. Is it typical for Julia packages to include benchmarks as part of the test suite, or within the same repo?

It will be interesting to see the results. There may be some overhead involving in calling functor because Julia functions must be invoked dynamically, by name, based on what operations appear in the expression tree. This seems to be unavoidable, though perhaps the implementation can be optimized. That said, it may not matter too much in practice because functor only has to be called once to produce an instance in LinearMaps or LinearOperators, which may then be called repeatedly. If Julia is smart about inlining one-liners, these instances should be just as fast as calling the underlying functions in LinearMaps or LinearOperators. Or so I hope.

[bosonbaas]

The only Julia package that I've seen that actually uses BenchmarkTools.jl is Krylov.jl, in which the benchmarks are kept in a folder called "benchmarks" located in the root folder. This is the expected location for benchmark files according to the PkgBenchmark.jl readme.

Do note that the PkgBenchmark.jl tool is only meant to be used to compare runtime of package functions across git commits, not between functions in a single commit. So, following their schema may be useful in the long run for tracking overall performance of the package, but we'll need to add in some custom reports to display functor overhead specifically.

[epatters]

I created #328 to track the set up of package benchmarks generally. I'll leave this issue open for now, since it would be still good to benchmark the overhead associated with functor.