search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

OpenJ9 Perf Results with OJDK MHs enabled #12728

Open babsingh opened 3 years ago

babsingh commented 3 years ago

This issue contains performance data for OpenJ9 with OJDK MHs enabled.

Target benchmarks

1) JRuby 2) Nashorn 3) scala-java-records-benchmark (https://github.com/harpocrates/scala-java-records-benchmark, https://github.com/eclipse-openj9/openj9/issues/12359)

nbhuiyan commented 3 years ago

For reference, here are the recent results of jruby and nashorn benchmarks on JDK11 comparing the performance of OpenJ9 with the existing OpenJ9 MethodHandle implementation vs the reference OpenJDK implementation (RI). The diff (performance gap) is calculated using the following: abs(OpenJ9 - RI)/RI * 100.

Jruby

headius/bench2018

red-black (lower=better)

OpenJ9 RI diff
1.9284 0.8336 56.77%

hash (higher = better)

OpenJ9 RI diff
bm_bighash 85.055 53.727 58.31%
bm_hash_long 54.22 55.655 2.58%
bm_hash_small2 188.857 184.896 2.14%
bm_hash_small4 146.211 137.465 6.36%
bm_hash_small8 93.796 76.529 22.56%

mandelbrot (lower=better)

OpenJ9 RI diff
3.9179 1.2743 67.47%

kwargs (lower=better)

OpenJ9 RI diff
2.7632 3.2769 18.59%

weird_sort (higher=better)

OpenJ9 RI diff
normal 15.363 90.886 83.1%
weird 10.117 14.225 28.88%

Nashorn (lower=better)

5371

OpenJ9 RI diff
closureTest 144328.249 56421.089 155.81%
javaTest 697020.8 446143.001 56.23%
loadJS 696068092.611 539940873.905 28.92%
mathTest 191154280.676 43922475.21 335.21%
objectTest 329148.44 184116.502 78.77%
regexTest 34280233.376 30393064.373 12.79%
babsingh commented 3 years ago

These results are collected using scala-java-records-benchmark. They were collected on 19 May 21. Lower score refers to better performance. The unit of score is ns/op [time/operation]. OpenJ9 w/ OJDK MHs + JIT still underperforms. There are pending changes, which will eventually improve the performance.

Benchmark OJDK JDK16 OpenJ9 JDK16 OpenJ9 w/ OJDK MHs OJDK JDK16 OpenJ9 JDK16 OpenJ9 w/ OJDK MHs
-Xint -Xint -Xint With JIT With JIT With JIT
Bench.largeJavaEqualityTest 51392.904 10339.209 23231.945 15.687 206.891 20975.193
Bench.largeJavaInequalityTest 4770.602 1204.771 2356.366 5.297 47.955 2053.796
Bench.largeScalaEqualityTest 5049.162 1903.295 1910.793 50.379 49.278 42.252
Bench.largeScalaInequalityTest 1874.411 678.598 689.985 10.063 10.135 9.709
Bench.smallJavaEqualityTest 9199.715 2018.07 4463.956 5.676 47.39 3637.156
Bench.smallJavaInequalityTest 4750.179 1163.417 2363.743 5.167 40.946 2012.29
Bench.smallScalaEqualityTest 1029.631 403.254 409.576 10.491 16.898 16.152
Bench.smallScalaInequalityTest 771.266 311.534 317.501 4.673 3.789 3.747
------------------------------ ---------- ------------ ------------------ ---------- ------------ ------------------
Total time [s] 1016 1123 1149 943 1002 1011
babsingh commented 3 years ago

@DanHeidinga More data will be reported next week with changes that will lead to better performance.

0xdaryl commented 3 years ago

It is unlikely we will have results with JIT enabled until late next week at the earliest. Once @jdmpapin's inliner work is enabled we can re-bench, but more analysis will likely be needed to identify further opportuntities.

ThanHenderson commented 1 year ago

Nashorn JDK11

Setup

nashorn: downloaded from link at https://github.com/eclipse-openj9/openj9/issues/5371 OpenJDK sha: d4a63e5e5c09a2d8fde2e2d8af2735e8cea8d27d OpenJ9 sha: 4ba8f3d3239a48d5cc56ba807275e3442ec2b7a8 OMR sha: 94e44382f4c519811ee1d45bda8601e1a8e79506 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Pipeline-Build-Test-Personal/18880/ RI: https://jdk.java.net/java-se-ri/11-MR2

Raw values arens/op. diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. Lower raw values are better; negative diff% values suggest better performance for OpenJ9 with OJDK MHs.

The sig diff? columns signify whether the 99.9% confidence intervals cross.

Results

AVG OpenJ9 OpenJ9 w/ OJDK MHs RI OJDK,J9 diff% OJDK,RI diff%
closeureTest 73840.444 86418.566 33505.513 17.03 157.92
javaTest 389427.281 394481.533 223702.7 1.3 76.34
loadJS 470714203.3 452817083.2 354069919.2 -3.8 27.89
mathTest 91896466.28 139162061.7 11701126.6 51.43 1089.3
objectTest 208376.025 197504.317 98528.471 -5.22 100.45
regexTest 21318446.5 19692941.02 19906138.23 -7.62 -1.07
ERROR OJDK,J9 sig diff? OJDK,RI sig diff?
closeureTest 421.577 1513.031 1177.102 TRUE TRUE
javaTest 1773.48 15959.113 5953.699 FALSE TRUE
loadJS 17884425.07 12030141.92 17208810.34 FALSE TRUE
mathTest 400759.264 13710161.61 206223.082 TRUE TRUE
objectTest 16014.034 6481.453 1282.917 FALSE TRUE
regexTest 269241.567 515983.918 235774.228 TRUE FALSE
babsingh commented 1 year ago

closeureTest and mathTest have huge regressions with OJDK MHs enabled in OpenJ9. Also, the errors are large for some benchmarks. We might have to collect perf profiles for OpenJ9 vs OpenJ9 w/ OJDK MHs enabled. Let's discuss next week once we have numbers from the other benchmarks.

ThanHenderson commented 1 year ago

JRuby JDK11

Setup

OpenJDK sha: d4a63e5e5c09a2d8fde2e2d8af2735e8cea8d27d OpenJ9 sha: https://github.com/eclipse-openj9/openj9/commit/4ba8f3d3239a48d5cc56ba807275e3442ec2b7a8 OMR sha: 94e44382f4c519811ee1d45bda8601e1a8e79506 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Pipeline-Build-Test-Personal/18880/ RI: https://jdk.java.net/java-se-ri/11-MR2 Benchmarks: https://github.com/headius/bench2018 with jruby-9.2.0.0 and the JDKs above, all other options set to empty.

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are IPS (Iterations per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

run_all.rb was invoked 6 times. For tests reporting IPS, the harmonic mean was used to calculate the overall mean. For tests reporting time, the arithmetic mean was used to calculate the overall mean.

Openj9 OpenJ9 w/ OJDK MHs RI OJDK,J9 diff% OJDK,RI diff% Higher is Better?
aref 23.52945362 24.50699197 36.62848373 4.15 -33.09 TRUE
kwargs 1.957237 1.848852225 1.665217558 -5.54 11.03 FALSE
mandelbrot 4.24541435 3.471999567 3.095899842 -18.22 12.15 FALSE
red_black 2.989383042 1.839528467 1.184561392 -7.53 55.29 FALSE
hash TRUE
bm_bighash 159.1381292 130.014898 172.5808049 -18.3 -24.66
bm_long 75.61861284 72.55878889 83.83419221 -4.05 -13.45
bm_small2 196.5139465 196.8587689 307.9008781 0.18 -36.06
bm_small4 150.2621203 129.8387028 194.5893566 -13.59 -33.28
bm_small8 101.1572531 81.9990624 130.0253759 -18.94 -36.94
weird_sort TRUE
normal 18.35367679 16.1438805 33.99895687 -12.04 -52.52
weird 8.499955682 9.731477391 11.68828367 14.49 -16.74
ThanHenderson commented 11 months ago

Evaluation after OJDK MH backport to JDK 11 and 8

Setup

JDK8

OpenJDK sha: 9a702f6d35a13affd72c1b996ca991ae40dac100 OpenJ9 sha: https://github.com/eclipse-openj9/openj9/commit/4ba8f3d3239a48d5cc56ba807275e3442ec2b7a8 OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK8_x86-64_linux_Personal/2945/ RI: https://jdk.java.net/java-se-ri/8-MR5

JDK11

OpenJDK sha: f229543b686dd32abe2b3f5565a46df0d70414b9 OpenJ9 sha: 58dc933a2633b2e24a7619c465e282fe5ed63418 OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK11_x86-64_linux_Personal/5956/ RI: https://jdk.java.net/java-se-ri/11-MR2

JDK17

OpenJDK sha: 549540814d003296474c332e25c5f109ca218cd2

OpenJ9 sha: 21f742fbc5756e350830a53d78dd3f4e8e48fba1 OMR sha:137778af49aa93a5780b8eebb9d688f4be8e47ac OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK17_x86-64_linux_Personal/1773/ RI: https://jdk.java.net/java-se-ri/17# JDK17

JRuby

Benchmarks: https://github.com/headius/bench2018 with jruby-9.2.0.0 and the JDKs above, all other options set to empty.

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are IPS (Iterations per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

run_all.rb was invoked 3 times. For tests reporting IPS, the harmonic mean was used to calculate the overall mean. For tests reporting time, the arithmetic mean was used to calculate the overall mean.

JRuby: JDK8

OpenJ9 OpenJ9 (OJDK MHs) RI OJDK MHs,J9 diff% OJDK MHs,RI diff% Higher is Better?
aref 25.449 22.904 36.198 -10 -36.72 TRUE
kwargs 1.8678 1.9312 1.6136 3.39 19.69 FALSE
mandelbrot 4.2291 3.3686 2.7213 -20.35 23.79 FALSE
red_black 2.0926 1.8215 1.1107 -12.95 64 FALSE
hash TRUE
bm_bighash 152.31 159.80 224.42 4.92 -28.79
bm_long 72.250 75.542 89.592 4.56 -15.68
bm_small2 186.38 235.23 384.75 26.21 -38.86
bm_small4 143.63 177.04 283.62 23.26 -37.58
bm_small8 89.667 98.510 165.02 9.86 -40.31
weird_sort TRUE
normal 19.865 16.857 32.185 -15.14 -47.63
weird 7.9614 9.9400 11.120 24.85 -10.62

JRuby: JDK11

OpenJ9 OpenJ9 (OJDK MHs) RI OJDK MHs,J9 diff% OJDK MHs,RI diff% Higher is Better?
aref 23.948 24.851 32.318 3.77 -23.1 TRUE
kwargs 1.8752 1.8618 1.6800 -0.71 10.82 FALSE
mandelbrot 4.1803 3.3257 3.1520 -20.44 5.51 FALSE
red_black 1.8992 1.8616 1.1793 -1.98 57.85 FALSE
hash TRUE
bm_bighash 160.82 137.41 177.96 -14.55 -22.78
bm_long 74.798 74.146 84.290 -0.87 -12.03
bm_small2 190.76 217.21 312.24 13.86 -30.43
bm_small4 140.41 146.92 192.44 4.64 -23.65
bm_small8 90.429 88.623 130.46 -2 -32.07
weird_sort TRUE
normal 15.832 17.277 32.451 9.13 -46.76
weird 8.2806 9.4855 11.605 14.55 -18.26

JRuby: JDK17

OpenJ9 (OJDK MHs) RI diff% Higher is Better?
aref 23.937 39.001 -38.62 TRUE
kwargs 1.9658 1.5349 28.08 FALSE
mandelbrot 3.9489 2.7269 44.81 FALSE
red_black 2.0294 1.1698 73.47 FALSE
hash TRUE
bm_bighash 156.52 197.97 -20.94
bm_long 75.412 87.625 -13.94
bm_small2 224.50 298.25 -24.73
bm_small4 192.11 210.06 -8.55
bm_small8 120.63 131.68 -8.39
weird_sort TRUE
normal 18.674 36.577 -48.95
weird 8.8396 13.259 -33.33

Nashorn

nashorn: downloaded from link at https://github.com/eclipse-openj9/openj9/issues/5371

Raw values arens/op. diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. Lower raw values are better; negative diff% values suggest better performance for OpenJ9 with OJDK MHs.

The sig diff? columns signify whether the 99.9% confidence intervals cross.

Nashorn: JDK8

AVG OpenJ9 OpenJ9 (OJDK MHs) RI OJDK MHs,J9 diff% OJDK MHs,RI diff%
closeureTest 91505.784 93466.824 54333.724 2.14 72.02
javaTest 560854.387 563618.939 351041.973 0.49 60.56
loadJS 316862499.5 336529749.7 313854511.7 6.21 7.22
mathTest 73244139.44 76200520.01 22154277.01 4.04 243.95
objectTest 161927.127 149042.848 64201.087 -7.96 132.15
regexTest 20030463.34 17915025.36 23130880.73 -10.56 -22.55
ERROR OJDK MHs/J9 sig diff? OJDK MHs/RI sig diff?
closeureTest 776.574 1307.805 1020.998 FALSE TRUE
javaTest 7143.667 7581.127 7808.42 FALSE TRUE
loadJS 7894732.838 11297730.9 18182676.45 TRUE FALSE
mathTest 1480766.703 1069644.212 660128.421 TRUE TRUE
objectTest 2467.68 2115.606 1193.525 TRUE TRUE
regexTest 172951.024 252756.481 251662.393 TRUE TRUE

Nashorn: JDK11

AVG OpenJ9 OpenJ9 (OJDK MHs) RI OJDK MHs,J9 diff% OJDK MHs,RI diff%
closeureTest 76699.613 79532.282 38031.226 3.69 109.12
javaTest 394880.844 388692.038 231436.955 -1.57 67.95
loadJS 307194678.5 299091809.6 243569415.8 -2.64 22.8
mathTest 74596025.51 75703800.27 17827959.42 1.49 324.64
objectTest 204520.483 175946.441 100407.056 -13.97 75.23
regexTest 20824044.65 19494917.04 19827037.09 -6.38 -1.68
ERROR OJDK MHs/J9 sig diff? OJDK MHs/RI sig diff?
closeureTest 1173.467 1261.971 2431.991 TRUE TRUE
javaTest 16818.449 8550.381 2424.928 FALSE TRUE
loadJS 9896791.459 9993521.638 12978266.56 FALSE TRUE
mathTest 1445430.885 3010597.129 278380.344 FALSE TRUE
objectTest 8965.226 2438.185 995.478 TRUE TRUE
regexTest 385095.336 587363.531 206990.58 TRUE FALSE

Nashorn: JDK17

AVG OpenJ9 (OJDK MHs) RI J9,RI diff%
closeureTest 77230.321 31929.544 141.88
javaTest 406561.527 210242.633 93.38
loadJS 317260120.4 190994449.7 66.11
mathTest 82003116.31 16345537 401.69
objectTest 209020.914 101615.085 105.7
regexTest 20157479.69 17433295.22 15.63
ERROR J9/RI sig diff?
closeureTest 991.429 1548.524 TRUE
javaTest 9544.566 3124.021 TRUE
loadJS 8859952.78 11497125.36 TRUE
mathTest 3268697.686 480921.858 TRUE
objectTest 11150.499 1281.714 TRUE
regexTest 189672.389 328671.301 TRUE
ThanHenderson commented 11 months ago

DayTrader7

Setup

JDK8

OpenJDK sha: 9a702f6d35a13affd72c1b996ca991ae40dac100 OpenJ9 sha: https://github.com/eclipse-openj9/openj9/commit/4ba8f3d3239a48d5cc56ba807275e3442ec2b7a8 OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK8_x86-64_linux_Personal/2945/

JDK11

OpenJDK sha: f229543b686dd32abe2b3f5565a46df0d70414b9 OpenJ9 sha: 58dc933a2633b2e24a7619c465e282fe5ed63418 OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244 OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK11_x86-64_linux_Personal/5956/

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are TPS (Transactions per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

JDK8 OpenJ9 OpenJ9 (OJDK MHs) diff% Higher == Better?
Startup 3554.76 3615.738 1.72 FALSE
Throughput 4758.766 4735.5 -0.49 TRUE
Throughput (warm) 4828.133 4701.583 -2.62 TRUE
JDK11 OpenJ9 OpenJ9 (OJDK MHs) diff% Higher == Better?
Startup 3824.976 3905.571 2.11 FALSE
Throughput 4764.75 4781.333 0.35 TRUE
Throughput (warm) 4810.017 4788.133 -0.45 TRUE
vijaysun-omr commented 10 months ago

I think we will need confirmation from (say) Marius Pirvu on the exact throughput situation here for Daytrader7. For example, I would certainly be reluctant to accept 2.62% throughout loss in Daytrader7 warm SCC runs in Java8 on x86 (it is not easy to get that kind of improvement in these kinds of apps being the reason). It is very possible that there is some fluctuation in the score and in fact, there isn't any substantial regression. So we just need to be more certain and debug to reduce the impact if the regression amount is in fact confirmed.

fyi @0xdaryl and @hzongaro if you have thoughts.

singh264 commented 4 months ago

@babsingh how can I know the status of the issue as it seems like it is a pre-requisite for https://github.com/eclipse-openj9/openj9/issues/18400 and it is important to document the thoughts?