Open hendriknielaender opened 6 months ago
One possible continuous benchmarking system that might be suitable is https://bencher.dev/, I've been poking around trying to get it running as a self-hosted service on a VPS, I'll see how it goes. It also has a paid cloud service.
that looks cool, i'll have a look at the docs 👍
I think what we can do here is to enhance the current json export, to support different JSON benchmark reports like the BMF JSON from https://bencher.dev/docs/explanation/adapters/#-json.
Or for this github action (benchmark-action), the are also supporting custom json.
Maybe we can also think about contributing to these projects to support our reports natively, if there are some benefits.
Another one that i found, but they do not offer a JSON spec. https://codspeed.io/
It looks like currently codspeed only supports python, rust, and node, but I guess they'll add more support as time goes on.
The way they benchmark seems a bit weird to me, they seem to focus on CPU time exclusively, so something that hardly uses the CPU but spends lots of unnecessary time waiting on IO would benchmark better than something that does a little more work on the CPU but reduces the IO load and reduces the overall run time. Unless I've misunderstood of course.
Summary
Introduce a GitHub Action specifically designed for continuous benchmarking using zBench. This action will automate the process of running benchmarks on every commit/pull request, collecting performance data, and visualizing the results over time. This feature aims to help maintainers and contributors monitor and improve the performance of their Zig projects consistently.
Background
Performance is a critical aspect of software development, and zBench offers a robust benchmarking tool for the Zig programming language. However, there's a gap in automatically monitoring these benchmarks over the development cycle. Integrating zBench with GitHub Actions provides an automated solution to fill this gap, making it easier to track performance changes and prevent regressions.
Proposed Features
Implementation Notes
Potential Challenges
Setup
.github/workflows
directory. An example workflow file is provided below.Example Workflow
Create a file
.github/workflows/zbench-benchmark.yml
with the following content:Conclusion
Integrating continuous benchmarking into the development process can significantly enhance the performance awareness and optimization efforts of Zig projects. By developing a dedicated GitHub Action for zBench, we can automate this process, providing developers with the tools they need to monitor and improve performance continuously.