Automation of Benchmark Runs for Improved Performance Tracking

[antimora]

It has been proposed to automate the running of benchmarks for improved tracking of performance changes over time. Currently, we have benchmarks (as seen in burn-benches) but these are manually run, which is not an ideal situation for continuous monitoring and comparison of changes in performance.

One of the main challenges highlighted is the availability of the appropriate hardware to test different backends. Our Continuous Integration (CI) process currently makes use of the machines provided by GitHub, which is somewhat limited for conducting comprehensive benchmarks.

A potential workaround could be limiting the tests to CPU devices on the provided backends. However, this may not give us a complete picture of the system's performance across different hardware setups.

Potential solutions to address this issue. For instance, we could look into:

• Integration with additional CI/CD tools that can provide more hardware diversity.
• Addition of partial, yet significant benchmarks that can run on GitHub-provided machines.
• Creation of a dedicated benchmarking suite which can run on a variety of devices and environments.

[mmalczak]

I think that the easiest solution is to add a dedicated Github Actions runner. It could be a self-hosted machine or on a cloud. The only limitation here is the price. However, it could be done even on some main developers laptop. The benchmarks could be run only at night, when these machines are not used anyway. I know that it is not a perfect solution, but it is almost cost-free (the only cost would be power consumption), very easy to do and could provide additional insight into performance.

[jggc]

I'm in, I have hardware sleeping right now. I'll take it up with @nathanielsimard to start coordinating this effort next week, he just poked me about this on Discord.