CSV/String parsing: Graal CE is slower, EE faster than Hotspot

rschwietzke commented 5 years ago

The attached code is from a project including several optimized version of it. This is all JMH setup. Graal CE RC14 runs it noticeably slower than without Graal and EE RC14 runs it really a lot faster. While I appreciate the EE performance and the understand the need for a commercial version, I think the CE version should not be slower than a default Hotspot.

Fun fact, JDK 11 shows a performance regression compared to JDK 8.

Code: jmh-xlt.zip

Summary

GraalVM EE 1.0.0-rc14

mvn clean compile install; java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseJVMCICompiler -jar target/benchmarks.jar CsvUtilsBenchmarkParse

Benchmark                              Mode  Cnt      Score      Error  Units
CsvUtilsBenchmarkParse.original        avgt    3  14720.133 ± 1472.599  ns/op
CsvUtilsBenchmarkParse.originalQuoted  avgt    3  14988.410 ± 2792.570  ns/op
CsvUtilsBenchmarkParse.tuned1          avgt    3  10391.310 ±  133.280  ns/op
CsvUtilsBenchmarkParse.tuned1Quoted    avgt    3  11063.147 ±  809.510  ns/op

GraalVM CE 1.0.0-rc14

mvn clean compile install; java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseJVMCICompiler -jar target/benchmarks.jar CsvUtilsBenchmarkParse

Benchmark                              Mode  Cnt      Score      Error  Units
CsvUtilsBenchmarkParse.original        avgt    3  17962.391 ± 7683.802  ns/op
CsvUtilsBenchmarkParse.originalQuoted  avgt    3  19840.872 ± 6191.967  ns/op
CsvUtilsBenchmarkParse.tuned1          avgt    3  15467.821 ± 8395.093  ns/op
CsvUtilsBenchmarkParse.tuned1Quoted    avgt    3  16242.020 ± 1016.787  ns/op

GraalVM EE 1.0.0-rc14; Graal DISABLED

EE and CE version deliver almost the same performance with disabled JVMCI Graal mvn clean compile install; java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:-UseJVMCICompiler -jar target/benchmarks.jar CsvUtilsBenchmarkParse

Benchmark                              Mode  Cnt      Score      Error  Units
CsvUtilsBenchmarkParse.original        avgt    3  15066.240 ± 2903.260  ns/op
CsvUtilsBenchmarkParse.originalQuoted  avgt    3  16119.593 ± 1148.570  ns/op
CsvUtilsBenchmarkParse.tuned1          avgt    3  12014.150 ±  450.277  ns/op
CsvUtilsBenchmarkParse.tuned1Quoted    avgt    3  12991.920 ±  308.249  ns/op

JDK 1.8.0_201, Java HotSpot(TM) 64-Bit Server VM, 25.201-b09

mvn clean compile install; java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -jar target/benchmarks.jar CsvUtilsBenchmarkParse

CsvUtilsBenchmarkParse.original        avgt    3  15804.135 ± 11706.042  ns/op
CsvUtilsBenchmarkParse.originalQuoted  avgt    3  16562.528 ±  9531.461  ns/op
CsvUtilsBenchmarkParse.tuned1          avgt    3  12979.102 ±  5872.863  ns/op
CsvUtilsBenchmarkParse.tuned1Quoted    avgt    3  13085.599 ±  2036.321  ns/op

JDK 11.0.2, OpenJDK 64-Bit Server VM, 11.0.2+9

Looks like a performance regression for JDK 11 over 8. mvn clean compile install; java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -jar target/benchmarks.jar CsvUtilsBenchmarkParse

Benchmark                              Mode  Cnt      Score      Error  Units
CsvUtilsBenchmarkParse.original        avgt    3  16685.595 ± 2187.195  ns/op
CsvUtilsBenchmarkParse.originalQuoted  avgt    3  17853.427 ±  774.100  ns/op
CsvUtilsBenchmarkParse.tuned1          avgt    3  13012.200 ± 1974.207  ns/op
CsvUtilsBenchmarkParse.tuned1Quoted    avgt    3  15603.583 ±  313.306  ns/op

john77eipe commented 5 years ago

That's a huge difference between CE and EE. It will be great if somebody needs to explain why this difference.

JohnTortugo commented 1 day ago

Can someone please add some labels to this issue? Perhaps compiler, performance, etc? Thanks

oracle / graal