adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
131 stars 314 forks source link

Identify and prioritize open perf benchmarks #1112

Open smlambert opened 5 years ago

smlambert commented 5 years ago

We want to select various (open-source) performance benchmark suites/harnesses to run at AdoptOpenJDK (for validating JDK binaries and to make available for easy use by JDK developers).

We can first list any that could be useful for JDK measurement/verification, identify why they are interesting, and then prioritize accordingly. This list can include both full suites and microbenchmarks with links to git repos (and pre-compiled jar/zip files when available).

Once we start adding test targets from different benches, we will also have to ensure that they are suitable for all implementations (not based on implementation specific classes, etc). But we can leave that as a separate exercise.

Larger suites HiBench: https://github.com/Intel-bigdata/HiBench

AcmeAir: https://github.com/blueperf/acmeair-monolithic-java https://acmeair.github.io/acmeair/

DayTrader3: https://github.com/WASdev/sample.daytrader3

DayTrader7: https://github.com/WASdev/sample.daytrader7

Dacapo: https://github.com/dacapobench/dacapobench http://dacapobench.org/

Micro/Macro Bench Harnesses Bumblebench: https://github.com/AdoptOpenJDK/bumblebench

jmh: https://openjdk.java.net/projects/code-tools/jmh/

Also look at https://github.com/Hyperfoil/

piyush286 commented 5 years ago

Thanks Shelley for creating this list! Besides the ones that you've already mentioned, here are some more open benchmarks:

Startup time, Footprint and Throughput of OpenLiberty with different apps: https://github.com/OpenLiberty/open-liberty + DT3, DT7, AcmeAir & More

Linkerd: https://github.com/linkerd/linkerd-examples/tree/master/perf-baseline/linkerd1-perf

Databricks: https://github.com/databricks/spark-perf

ES Rally: https://github.com/elastic/rally

Spring Boot Startup Benchmark: https://github.com/dsyer/spring-boot-startup-bench

FrameworkBenchmarks: https://github.com/TechEmpower/FrameworkBenchmarks

Perf Team's Microbenchmarks We've also written a few micro-benchmarks such as for crypto and GPU over the few years so we could think of adding those if you think they might be interesting and useful.

SPEC I guess we're just talking about open benchmarks here but we can also think of running some benchmarks from Standard Performance Evaluation Corporation (SPEC) such as SPECjbb2015 if it's possible to get licenses for them. It would be good to have them as part of AdoptOpenJDK quality assurance and possibly publish official results (https://www.spec.org/jbb2015/results/jbb2015.html) for our releases.

Priority I've already got started on adding Open Liberty with different apps (DT3, DT7 & AcmeAir) since that's the highest priority for us. I'm planning to schedule a meeting soon to discuss about perf testing. FYI @vijaysun-omr

smlambert commented 5 years ago

Thanks @piyush286 ! I have also added a note to the AdoptOpenJDK testing channel to solicit feedback from others (on what open benchmarks they choose, and why)... and looks like we have a nice selection already.

smlambert commented 5 years ago

I also agree that we should consider SPEC benchmarks. We will have to look at the licenses, to understand all of the details.

As I mentioned in our meeting, it could be that we start with the scenario that we add the metadata for running them (as special level), and treat it like JCKs, where folks with licenses could choose to run those targets, moving towards a scenario where we may license & publish results. Ultimately, we will want to look closely at the details, and decide if those benchmarks meet the criteria (open/transparent) and therefore fit into the AQA story or not.

jerboaa commented 5 years ago

https://renaissance.dev/

To quote their website:

Renaissance is a modern, open, and diversified benchmark suite for the JVM, aimed at testing JIT compilers, garbage collectors, profilers, analyzers and other tools.

smlambert commented 4 years ago

https://github.com/Hyperfoil/ also interesting

smlambert commented 3 years ago

Adding https://github.com/spring-projects/spring-petclinic to the list here, so that I can close related: https://github.com/AdoptOpenJDK/openjdk-tests/issues/1501 as we do not need to keep both issues open.

smlambert commented 2 years ago

Instead of tracking it via a separate issue, will close https://github.com/adoptium/aqa-tests/issues/1255 and track databricks and hibench via this issue.