Add placeholder ordo-benchmark and JMH benchmarks

helmetcrest commented 2 weeks ago

There's only one sample benchmark for each system at the moment, so as to keep the PR scope in check.

There are a couple of known issues:

I've heard that Gradle doesn't reliably rebuild the benchmark project when a change occurs to a dependency. Locally, I've had it skip running benchmarks when running gradlew jmh.
On Linux x86_64, swift package benchmark fails with a SIL verification failed: Two variables with different type but same scope! exception in the compiler. I can attach some logs if that'd be useful.

I'm not sure if this PR is too early to merge, but I'd very much appreciate pointers on how to address either the above issues, or anything else that seems unusual in my PR. For instance, my formatter keeps trying to fix indents; if that's too noisy, I can manually remove them for now and open a follow-up PR.

I can confirm that everything passes act pull_request, at least, but I needed to add a dependency on jemalloc in order to build the Swift benchmarking tool.

ktoso commented 2 weeks ago

This all looks excellent, thank you!

I only added a small note into the readme how to run the benchmarks :)

ktoso commented 2 weeks ago

I have now included building (not running) the benchmarks in CI as well so we don't regress them by accident.

ktoso commented 2 weeks ago

heh compiling swift benchmarks crashes on linux. I'll disable them in CI for now.

swiftlang / swift-java

Add placeholder ordo-benchmark and JMH benchmarks #134