Enhancing Java Library Template with Automated JMH Benchmark Integration

[sisimomo]

Describe the feature

In the spirit of fostering high-performance Java library development, I'd like to propose an enhancement to your Java library template: the integration of automated Java Microbenchmark Harness (JMH) benchmarks. By incorporating the "benchmark-action/github-action-benchmark" GitHub Action, we can offer an out-of-the-box solution for performance benchmarking in any library created from this template.

This integration would automatically run JMH benchmarks against pull requests, providing immediate feedback if a change negatively impacts performance beyond an acceptable threshold. It's a proactive measure to ensure that all libraries developed using your template adhere to the highest standards of performance from the outset. Not only does this streamline the development process for library maintainers, but it also instills a culture of performance awareness and optimization within the community.

Automating benchmarking through this GitHub Action eliminates the need for manual performance checks, making the library development process more efficient and reliable. It allows contributors to focus on innovation and functionality, secure in the knowledge that performance standards are being upheld automatically.

Additional information

• [x] Would you be willing to help implement this feature?

Final checks

• [X] Read the contribution guide.
• [X] Check existing discussions and issues.

[hartmut-co-uk]

Hi @sisimomo, apologies for the delayed response. This is an excellent idea to add to the template.

Could you briefly describe the solution you have in mind and the changes you plan to make to the Gradle project and the CI/CD workflow?

[sisimomo]

Hi @hartmut-co-uk,

I'm far from being an expert in CI/CD workflows or JMH Benchmarking, but here's the idea I was thinking about.

The idea is to use the GitHub Action, "benchmark-action/github-action-benchmark," configured to activate on pull requests. This action will automatically run the JMH benchmarks defined in our project against the changes in the pull request. Then, it compares these results with the benchmarks from the targetted branch to spot any performance regressions. Then, the action will post the results directly in the comments section of the pull request to ensure visibility.

Furthermore, we can set specific performance thresholds within the GitHub Action's configuration. If a pull request results in benchmark performance falling below these thresholds, the action can automatically flag the pull request for further review. We could create documentation that will provide guidelines on how to establish and adjust these thresholds to align with our library's performance objectives.

However, I'm not entirely sure how feasible this plan is. As mentioned in the Future Work section of the benchmark-action/github-action-benchmark project, there's only partial support for pull requests. Meanwhile, the Caveats section mostly discusses the impact of such workflows on GitHub pages. Although, I believe we won't need a GitHub page for testing results; having the results visible directly in the pull request should suffice.

Unfortunately, my free time is quite limited, making it difficult for me to tackle this task. However, I wanted to share the idea with you, so if you're interested, you might consider implementing it.

Anyway, thanks for this very nice template!

[hartmut-co-uk]

Many thanks, @sisimomo, for the detailed answer to my question. I like the idea, but not every project may use JMH, so I worried it's too much boilerplate.

🤔 It could be an initial opt-in feature the user would pick (ref https://github.com/thriving-dev/java-library-template/issues/33#issuecomment-1987716235). But that might not be trivial, and I'm worried about maintaining this long-term. There might have been better choices to provide selecting features compared to GitHub Templates.