search

scala / compiler-benchmark

Benchmarks for scalac
Other
68 stars 31 forks source link

benchmark JVM varieties (graal, jdk9, ...) #35

Open retronym opened 7 years ago

retronym commented 7 years ago

Our benchmarking infrastructure doesn't currently support benchmarking with alternative JVM without "polluting" the main data series. We need update it to add the contents of the system properties "java.vm.name" and "java.runtime.version" as tags that can be used for filtering in Grafana.

We also need to migrate historical data to populate the tags, as grafana makes it impossible (or at least really hard) to to the equivalent of getOrElse in the query to default absent tags.

retronym commented 7 years ago

Hacky manual attempt:

https://github.com/scala/scala/pull/6018

Results

retronym commented 7 years ago

First result looks great, better-files is 0.72x (!!!). scala appears on track for a similar speedup. Wow.

lrytz commented 7 years ago

Cool! That's measuring the warm VM I assume? Can you easily run a cold bench? As far as I remember, Graal took more effort to warm up in the past than HotSpot.

retronym commented 7 years ago

Will do when I get a chance. It "felt" equal or better given the timings of the 1st/2nd iteration of hot benchmark, though.

lrytz commented 7 years ago

IIRC the first iteration in our jmh benchmark already includes a number of executions, so there's quite some time for warmup already in there.