Notes

This commit rewrites our existing benchmarks and migrates them to JMH, integrated with our gradle build system using a plugin. We decided to migrate to JMH for a few reasons: reproducibility, isolation, accounting for JVM JIT optimizations, and (not least of which!) pretty graphs and configurable output format. For more details on the tradeoffs between conventional benchmarks and JMH, see this blog post.

Other than the benchmarks themselves, the real meat of this change lives in build.gradle. The JMH plugin introduced a few gradle tasks whose names collided with some of our existing tasks, so some existing task names were made more specific. The tricky parts of this change were modifying the JMH task sourceSets and JVM args to point to all the relevant source files in our codebase. Similarly, we had to add some static resources (as symlinks to avoid duplication) to a JMH resources directory where gradle can find them and bundle them into the benchmark JAR.

We also provide a ./bin/run-benchmarks.sh script to install dependencies (if needed), correctly invoke the JMH benchmarks, and generate a report.

The benchmarks currently compare ACCP against BouncyCastle and built-in Java providers. If we want to add BouncyCastle FIPS to the mix, it should be easy to add BCFIPS as a maven dependency for jmh as we did for vanilla BouncyCastle then include it in our providers lists in the individual benchmarks.

These JMH changes were backported to ACCP 1.6 on the benchmarking-1.x branch for historical comparison. You can see what the benchmark reports look like here:

The improvements from ACCP 1.6 (on the left, backed by OpenSSL 1.1.1) to 2.0 (on the right, backed by AWS-LC) are particularly dramatic for elliptic curve algorithms:

Screen Shot 2023-01-30 at 11 39 32 AM Screen Shot 2023-01-30 at 11 40 08 AM

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

WillChilds-Klein commented 1 year ago

merging. please feel free to leave additional comments on this PR if you have opinions on benchmark parameters (e.g. benchmark duation, # iterations, etc.) or additional use-cases/patterns to benchmark.

SalusaSecondus commented 1 year ago

The benchmarks in this repo should probably be entirely thrown away and rewritten. They are ancient and were never tuned or updated.

corretto / amazon-corretto-crypto-provider

Overhaul benchmarks, migrate to JMH #263

Notes