search

eclipse-ocl / org.eclipse.ocl

Eclipse Public License 2.0
0 stars 0 forks source link

[cg] Investigate 'slower' Java 11 performance #2126

Open eclipse-ocl-bot opened 1 week ago

eclipse-ocl-bot commented 1 week ago

| --- | --- | | Bugzilla Link | 565563 | | Status | NEW | | Importance | P3 normal | | Reported | Jul 27, 2020 04:06 EDT | | Modified | Jul 28, 2020 02:43 EDT | | See also | 565555, 565558, 565368 | | Reporter | Ed Willink |

Description

When using Java 11, tests performance appears to be 2-3 times slower than Java 8. This is a subjective observation. Is it true? If so why?

Possible genuine factors

Possible confounding factors

NB. I also observe that my tower is slower than my laptop although both have similar RAM disk configs and I would expect the tower to be faster. Maybe a related confounding factor explanation.

Most of the confounding factors can be eliminated by comparing the Tycho stage elapsed time measurements on the Jenkins server. Multiple alternating runs should establish how much variation is pot luck wrt concurrent users on the same server. (Assumes that Java 8 and Java 11 have compatible and sensible installation on the server.)

eclipse-ocl-bot commented 1 week ago

By Ed Willink on Jul 27, 2020 07:15

ocl-branch-tests already set up as:

Java version: 12.0.2, vendor: Oracle Corporation, runtime: /opt/tools/java/openjdk/jdk-12/12.0.2+10\ Default locale: en_US, platform encoding: UTF-8\ OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"

Initial 'stable' builds fail with

Missing requirement: org.apache.lucene.analyzers-common 8.4.1.v20200122-1459 requires 'osgi.ee; (&(osgi.ee=JavaSE)(version=1.8))' but it could not be found

But the 'nightly' is ok. Ignore the above magic.

Three 12.0.2 runs give broadly similar results ~10% spread, with broad uniformity in the faster/slower.

org.eclipse.ocl.uml 5.10.0 0.470, 0.493, 0.561 s,\ org.eclipse.ocl.pivot 1.12.0: 7.161, 8.526, 9.890 s,\ org.eclipse.ocl.examples.xtext.tests 3.15.100: 05:20, 06:04, 05:46 min,\ Total 10:59, 12:24, 11:48 min

Switching to

Java version: 1.8.0_202, vendor: Oracle Corporation, runtime: /opt/tools/java/oracle/jdk-8/1.8.0_202/jre\ Default locale: en_US, platform encoding: UTF-8\ OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"

The nightly build fails with the original Bug 565555 org.eclipse.search problem for Java 11. The stable build works. Why didn't it work for Java 11 too?

org.eclipse.ocl.uml 5.10.0 0.400, 0.423, 0.444 s,\ org.eclipse.ocl.pivot 1.12.0: 5.409, 6.861, 6.348 s,\ org.eclipse.ocl.examples.xtext.tests 3.15.100: 04:51, 04:59, 04:59 min,\ Total 09:12, 09:39, 09:30 min

org.eclipse.ocl.uml involves just JDT compilation of Java 5 compatible source.

org.eclipse.ocl.pivot involves just JDT compilation of Java 8 compatible source with extensive use of @NonNull/@Nullable.

org.eclipse.ocl.examples.xtext.tests involves JUnit and quite a few auto-generated code generations/compilation and run.

There is a faint possibility that a new JDT/platform feature in the nearly 4.17M2 nightly used for Java 12 rather than the stable 4.16 for Java 8 has slowed things down - seems a bit unlikely.

It seems fair to conclude that for a JDT compiled Eclipse hosted workload, Open JDK Java 12 is 20% slower than Oracle Java 8. The original subjective observation of 2-3 times slower must involve a confounding factor.

The consistent differences for org.eclipse.ocl.uml compilations eliminate the possibility that the differences are caused by defective EMF/OCL/Xtext support for Java 12.

Since JUnit seems to experience similar ratios and we should be generating the same class files for Java 8/12, the difference must be a worse/thirstier JVM. Maybe a 32/64/128 bit creep.

Running the Java 12 builds again to reduce the possibility that the Java 8 runs hit a sweet spot in regard to concurrent builds...

Total time: 11:05, 11:10, 11:38 min

A bit more uniform, but no faster.

eclipse-ocl-bot commented 1 week ago

By Andrey Loskutov on Jul 27, 2020 08:13

(In reply to Ed Willink from comment #1)

ocl-branch-tests already set up as:

Java version: 12.0.2, vendor: Oracle Corporation, runtime: /opt/tools/java/openjdk/jdk-12/12.0.2+10

Please note, Java 12 is not supported since 2019: https://www.oracle.com/java/technologies/java-se-support-roadmap.html.

So I would recommend to use either Java 11 LTS or Java 14.

It seems fair to conclude that for a JDT compiled Eclipse hosted workload, Open JDK Java 12 is 20% slower than Oracle Java 8. The original subjective observation of 2-3 times slower must involve a confounding factor.

\ Have no idea what OCL is ding at all, but we saw a massive performance drop in some specific number formatting operations caused by a regression in JDK, that was just recently fixed https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8242108. I couldn't see if the regression was also on Java 12 branch or not, but you can try if running on Java 11.0.8 (that contains the patch) fixes anything.

eclipse-ocl-bot commented 1 week ago

By Andrey Loskutov on Jul 27, 2020 08:39

(In reply to Andrey Loskutov from comment #2)

I couldn't see if the regression was also on Java 12 branch or not

Performance regression we saw is not in 12.0.2.

So one should try to reduce OCL performance issue to a small standalone code snippet.

Have you tried to isolate the problem to a smaller test & profile that?\ Is this regression using 4.17 platform only, or also with 4.16?

eclipse-ocl-bot commented 1 week ago

By Ed Willink on Jul 27, 2020 08:59

(In reply to Andrey Loskutov from comment #3)

So one should try to reduce OCL performance issue to a small standalone code snippet.

As I wrote earlier:

"org.eclipse.ocl.uml involves just JDT compilation of Java 5 compatible source."

The measurement is a Tycho elapsed time for compilation only. Zero EMF/OCL/Xtext code is executed, so the only possible EMF/OCL/Xtext project influence could be an adverse programming style that affects Java 12 worse than Java 8, but since JDT is probably producing the same byte code from the same source code, it would seem that JDT itself can hardly be involved. It must be the JVM.

(In reply to Andrey Loskutov from comment #2)

Performance regression we saw is not in 12.0.2.

To be clear, my results cannot be dismissed just because of the massive rate of Java version churn invalidating measurements on Java 12.

This bug was raised to investigate a 2-3 fold subjective speed difference and fails to substantiate the subjective observation. It does however provide evidence that Java 12 is nearly 20% worse than Java 8.

More detailed work should be performed under the auspices of the platform project who should characterize each new release against each credible JVM on the SAME execution platform as the previous release. I look forward to seeing some performance curves.

eclipse-ocl-bot commented 1 week ago

By Andrey Loskutov on Jul 27, 2020 10:07

Ed, don't get me wrong, I want help you, not dismiss anything.\ The problem is however, that Java 12 is not supported anymore, and considering amount of Java bugs in non LTS Java releases I don't want to waste time hunting already fixed issues.

So can you please use either 11.0.8 JDK or latest 14 version to run your tests?

Regarding the tests: I'm a bit confused what exactly is slower. Is this test compilation or test execution times you are talking about? If it is only test compilation, you probably hit ct.sym performance issue in the ECJ batch compiler, see bug 563318. I don't know which compiler your maven build uses, but I assume it is ecj and not javac. You may try to use javac and check if that one is faster, or you may to compile OCL (for test only) with the Java 12 as target, to avoid compiler to run over ct.sym abstraction, would be interesting to see if thst fixes the performance.

If this is really related to ct.sym case, the problem is not that Java 12 is slower, but the way how java architects defimed https://openjdk.java.net/jeps/247. The way how it was done caused lot of troubles in JDT, because we have to re-implement what javac does without any formal public specification.

eclipse-ocl-bot commented 1 week ago

By Ed Willink on Jul 27, 2020 10:11

https://technology.amis.nl/2018/11/23/comparing-jvm-performance-zulu-openjdk-openjdk-oracle-jdk-graalvm-ce/#comment-9847 makes some very interesting observations on how the -Xmx setting can heavily influence OpenJDK GC performance making almost any like for like comparison suspect. It is very possible that legacy settings of Java options are failing to get the compatible best out of each new JVM. For these tests I made no attempt to change or even inspect the options.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=561908 identifies how JAVA_TOOL_OPTIONS has to be tweaked to get a Java 5 build to work, so there is a possibility that the default JIRO configuration is giving us sub-optimal settings.

Clearly GC tuning is another major confounding factor.

eclipse-ocl-bot commented 1 week ago

By Ed Willink on Jul 27, 2020 10:24

(In reply to Andrey Loskutov from comment #5)

Regarding the tests: I'm a bit confused what exactly is slower. Is this test compilation or test execution times you are talking about?

The org.eclipse.ocl.uml/ocl.eclipse.ocl.pivot measurements are for JDT compilation only. I think Tycho uses ecj. I doubt this varies between Java 8 and 12.

The org.eclipse.ocl.examples.xtext.tests measurements are primarily for test execution, but many of the tests auto-generate code which is compiled and tested as part of the test.

Pretty much everything is fairly consistently slower on Java 12.

After reading https://technology.amis.nl/2018/11/23/comparing-jvm-performance-zulu-openjdk-openjdk-oracle-jdk-graalvm-ce/#comment-9847 I consider my nearly 20% observation to be within the bounds of GC mis-tuning. So 20% slower is a valid observation for what the EF provides us as the JIRO defaults. It is not necessarily valid in any other environment, certainly not in one that has been carefully tuned. Java 11 might be genuinely different to Java 12 or just subject to different tuning. I don't have time or motivation to investigate the 20% further.

I remain interested in substantiating / rubbishing the original 2-3 fold subjective observation.

eclipse-ocl-bot commented 1 week ago

By Andrey Loskutov on Jul 27, 2020 10:29

(In reply to Ed Willink from comment #6)

Clearly GC tuning is another major confounding factor.

Yes, this reminds me on another subtle issue we've had with compilation on Java 11: the default GC under Java 11+ is G1 (not the parallel GC!), and G1 by default is NOT configured for throughput one needs on command line compilation! You have to use "-XX:GCTimeRatio=99" to enforce that application should get at least 99 times as much time as the collector. The default in Java 11 is 12, in Java 8 is 99.

This was another tweak (after ct.sym fix) we needed in our product to get the compile times under Java 11 comparable with times under Java 8 (we also saw ~10% degradation in compilation times).

eclipse-ocl-bot commented 1 week ago

By Ed Willink on Jul 28, 2020 02:43

(In reply to Ed Willink from comment #6)

Clearly GC tuning is another major confounding factor.

No; just a potential confounding factor if a comparison is made where one is tuned and the other is not. The results here are for an out of the box comparison of EF-hosted Eclipse-based JDT compilations and executions.

If a a particular Java is badly tuned out of the box, then that bad tuning can justifiably be considered as intrinsic. Java 12 is slower for the test workload, albeit there is an opportunity for skilled users to mitigate.