Some benchmarks for comparing the Java and Kotlin versions of the parser

[psfblair]

This is pretty unscientific, but I added a test to do a hundred runs of each of the unit tests to gauge performance, so as to compare it with the Kotlin version. I found the performance highly variable, probably because my laptop is doing other things; the only real way to do this would be to put it on a dedicated host. But at least it gives a ballpark idea. The Kotlin version's performance is pretty close to the Java performance, and the variability in the two test runs was much bigger than the difference between the two parsers.

Here are the comparisons:

MacBook Air (Early 2015), 2.2 GHz Intel Core i7, 8 GB 1600 MHz DDR3 RAM macOS 10.13.6 IntelliJ IDEA 2020.1

Java benchmark: (Results in seconds) ===================== RESULTS FOR arithmetic===================== DoubleSummaryStatistics{count=100, sum=1.354179, min=0.004290, average=0.013542, max=0.267944}

===================== RESULTS FOR java-grammar===================== DoubleSummaryStatistics{count=100, sum=62.343267, min=0.533682, average=0.623433, max=0.881978}

===================== RESULTS FOR java-parse===================== DoubleSummaryStatistics{count=100, sum=99.580973, min=0.784130, average=0.995810, max=1.311033}

Kotlin benchmark:

(Results in seconds) ===================== RESULTS FOR arithmetic===================== DoubleSummaryStatistics{count=100, sum=1.394441, min=0.004973, average=0.013944, max=0.252265}

===================== RESULTS FOR java-grammar===================== DoubleSummaryStatistics{count=100, sum=63.733993, min=0.499645, average=0.637340, max=1.354979}

===================== RESULTS FOR java-parse===================== DoubleSummaryStatistics{count=100, sum=103.694866, min=0.826100, average=1.036949, max=1.479693}

---

Some time later, another comparison:

Second Java benchmark, later:

===================== RESULTS FOR arithmetic===================== DoubleSummaryStatistics{count=100, sum=1.843691, min=0.004564, average=0.018437, max=0.329759}

===================== RESULTS FOR java-grammar===================== DoubleSummaryStatistics{count=100, sum=71.491856, min=0.601802, average=0.714919, max=1.022379}

===================== RESULTS FOR java-parse===================== DoubleSummaryStatistics{count=100, sum=111.961126, min=0.884459, average=1.119611, max=1.496312}

Kotlin benchmark: ===================== RESULTS FOR arithmetic===================== DoubleSummaryStatistics{count=100, sum=1.788753, min=0.005947, average=0.017888, max=0.192602}

===================== RESULTS FOR java-grammar===================== DoubleSummaryStatistics{count=100, sum=73.348304, min=0.578634, average=0.733483, max=1.196941}

===================== RESULTS FOR java-parse===================== DoubleSummaryStatistics{count=100, sum=113.870696, min=0.920826, average=1.138707, max=1.393449}

[lukehutch]

Good to know the Kotlin parser now runs at approximately the same speed!

[lukehutch]

BTW Java gives extremely uneven runtimes, even for fully deterministic programs. It's not usually just an issue of other CPU loads interfering with a benchmark. Garbage Collection and the JIT also seem to work in a highly unpredictable way.