Closed gdams closed 4 years ago
Phase 1 ready for publishing (Linux x64, Mac x64, Windows x64/x86). https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/704/
+1
Test failures:
Mac: extended.system: HCRLateAttachWorkload_0 => Failed on 4/5 last builds. Raised openjdk-tests/1555 - looks like a long-standing, recurring issue (as seen in deep history view)
Linux sanity.openjdk: java/io/StringBufferInputStream/OverflowInRead.java.OverflowInRead => Passed on re-run Grinder/1772 java/math/BigInteger/LargeValueExceptions.java.LargeValueExceptions => Passed on re-run Grinder/1774 java/net/Inet6Address/B6206527.java.B6206527 => Intermittent (issue) Passed on re-run Grinder/1775 java/net/ipv6tests/B6521014.java.B6521014 => Intermittent (issue) Passed on re-run Grinder/1776
special.function: MBCS_Tests_CLDR_11_ja_JP_linux_0 MBCS_Tests_CLDR_11_ko_KR_linux_0
Linux sanity.openjdk: java/io/StringBufferInputStream/OverflowInRead.java.OverflowInRead java/math/BigInteger/LargeValueExceptions.java.LargeValueExceptions java/net/Inet6Address/B6206527.java.B6206527 java/net/ipv6tests/B6521014.java.B6521014
FWIW, these passed for the upstream build: https://ci.adoptopenjdk.net/view/Test_upstream/job/Test_openjdk11_hs_sanity.openjdk_x86-64_linux_upstream/33/
We should run grinders on those, using the same machines (LABEL=hostname) that the upstream jobs ran on, to rule out machine issues.
@smlambert I tried doing that but I think I set the job up wrong? https://ci.adoptopenjdk.net/job/Grinder/1768/console
You need TARGET=jdk_custom and LABEL=machine name
Oh and sorry, CUSTOM_TARGET needs to be the full path with 'test/jdk' for 11+ at the front and no OverflowInRead at the end eg. test/jdk/java/io/StringBufferInputStream/OverflowInRead.java
https://ci.adoptopenjdk.net/job/Grinder/1771/ runs the correct TARGET and CUSTOM_TARGET values, though I did not restrict it to a particular machine by setting LABEL=hostname
Shelley you still have the .OverflowInRead
at the end
The 6 failing test targets have been triaged in https://github.com/AdoptOpenJDK/TSC/issues/132#issuecomment-574722811 and shown to be non-blocking failures, +1 to publish.
Releasing Phase 1 platforms (Linux x64, Mac x64, Windows x64/x86)
Phase 2 pipeline complete: https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/706/
Test failures:
AIX: extended.system: LockingLoadTest_0 MathLoadTest_all_0
sanity.openjdk: java/lang/ProcessBuilder/Basic.java => (Fails Intermittently https://github.com/AdoptOpenJDK/openjdk-tests/issues/1013, Upstream Bug Report) java/nio/channels/AsyncCloseAndInterrupt.java => Known to fail intermitently on Linux but NOT AIX (https://github.com/AdoptOpenJDK/openjdk-tests/issues/602)
Linux - aarch64: => All intermittently fail on upstream too (see https://github.com/AdoptOpenJDK/TSC/issues/132#issuecomment-575089217) java/lang/invoke/VarHandles/VarHandleTestAccessBoolean.java java/lang/invoke/VarHandles/VarHandleTestAccessDouble.java java/lang/invoke/VarHandles/VarHandleTestAccessFloat.java java/lang/invoke/VarHandles/VarHandleTestAccessShort.java java/lang/invoke/VarHandles/VarHandleTestAccessString.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessByte.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessChar.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessDouble.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessFloat.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessInt.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessLong.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessShort.java
Linux - aarch64: java/lang/invoke/VarHandles/VarHandleTestAccessBoolean.java java/lang/invoke/VarHandles/VarHandleTestAccessDouble.java java/lang/invoke/VarHandles/VarHandleTestAccessFloat.java java/lang/invoke/VarHandles/VarHandleTestAccessShort.java java/lang/invoke/VarHandles/VarHandleTestAccessString.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessByte.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessChar.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessDouble.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessFloat.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessInt.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessLong.java java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessShort.java
We've been seeing those VarHandle test failures on Aarch64 for upstream builds intermittently too. We are going back and forth about this as we a) would need to run jcstress on that hardware to know whether or not these are HW bugs. b) would need access to those machines in order to see if failures reproduce when being run manually. It's yet unclear whether these are actual bugs (in HW or hotspot code).
On known good hardware we (internally) don't see those failures.
The extended.system failures on AIX need further investigation. I think they are the same failures that @karianna reported on last release (that we had a follow-up action to look at - but did not complete that action: https://github.com/AdoptOpenJDK/TSC/issues/119#issuecomment-566146382).
Investigation of AIX failures: launch the individual test targets in a Grinder against the past releases to see if the failures are seen in those releases.
MathLoadTest_all : Looks like this has existed for a while all the way back to April 2019 release, grabbed https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.3%2B7/OpenJDK11U-jdk_ppc64_aix_hotspot_11.0.3_7.tar.gz and see same failure https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1792
LockingLoadTest uses some of the Math API tests underneath, and is failing with the same error.
LLT testFailure: testTan(net.adoptopenjdk.test.math.MathAPITest): tan(double)[10] :: expected:<1.2786058599594134> but was:<1.2786058599594137>
LLT junit.framework.AssertionFailedError: tan(double)[10] :: expected:<1.2786058599594134> but was:<1.2786058599594137>
LLT at junit.framework.Assert.fail(Assert.java:57)
Verified that it has existed at least as far back as the April 2019 release (Grinder 1793).
Given we have already released with this failure, will not block this release due to that failure (in MathAPITest), but recommend a deeper investigation to see why it passes on openj9 but fails on hotspot (and determine if its a test case that needs to be fixed or a long-standing 'real' issue).
I am now looking at OpenJ9 jdk11 pipelines launched by @lumpfish:
Submitted openj9 jdk11 builds as: https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/708/ - xlinux (built with new jitserver feature which requires and extra build configure arg)
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/710/ - xlinuxXL (built with jitserver - had to resubmit owing to problem with build machine)
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/709/ - all other platforms except AIX.
AIX builds are currently taking very long to run, so don't want to hold up releasing the other platforms https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/711/ - AIX . (waiting on machine)
Pipeline 708: same 2 test targets failing from special.functional group as did in the hotspot build.
MBCS_Tests_CLDR_11_ja_JP_linux_0
MBCS_Tests_CLDR_11_ko_KR_linux_0
I believe this to be a test case issue that was previously reported (and thought to be fixed).
https://github.com/AdoptOpenJDK/openjdk-tests/issues/1394 and fixed: https://github.com/AdoptOpenJDK/openjdk-tests/pull/1406 but we should raise an issue (or reopen 1394) for these 2 failing targets.
Pipeline 709: only 2 test targets failing from sanity.openjdk against mac_xl build (not seen on mac build):
jdk_lang: java/lang/invoke/lambda/LogGeneratedClassesTest.java - seen this before (reported in various openj9 issues, and openjdk-tests/585) - Grinder 1794 passes
Deep history for jdk_lang on jdk11 j9 mac_xl has been green for a long time, so if intermittent, very infrequent.
jdk_rmi: java/rmi/activation/Activatable/checkRegisterInLog/CheckRegisterInLog.java - seen this one before (reported in past issues openj9/4772, openjdk-tests/585) - Grinder 1795 passes
Deep history on jdk_rmi on jdk11 j9 mac_xl also green.
Ran some Grinders (links above) which passed. Deep history shows no previous failures. Will not block on these 2 test failures, though recommend grinder with ITERATIONS=xx to see if can reproduce (perhaps on same machine where failures seen in release build).
Pipeline 710: same 2 test targets failing from special.functional group as did in the hotspot build.
MBCS_Tests_CLDR_11_ja_JP_linux_0
MBCS_Tests_CLDR_11_ko_KR_linux_0
Considered non-blocking.
@smlambert am I good to ship the secondary platforms for jdk11 then?
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/710/ - xlinuxXL (built with jitserver - had to resubmit owing to problem with build machine)
Pipeline job 710 is an xLinux-openj9 build not xLinuxXL-openj9
jdk-11.0.6+10_openj9-0.18.0 for xLinuxXL being resubmitted, as was misstakenly re-submitted as xLinux last time: Queued: https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/713/
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/711/ - AIX: Failed to build: Sjavac server failed to initialize: No port file values materialized. Giving up after 60195 ms Will wait for @sxa555 to analyse
jdk-11.0.6+10_openj9-0.18.0 ppc64leLinuxXL sanity.openjdk job in pipeline 709 failed: https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_xl/37/. re-ran a grinder: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1799/
jdk-11.0.6+10_openj9-0.18.0 summary so far:
Pipelines Ready for Publish, no blocking failures:
Waiting:
pLinuxXL-openj9 openjdk.sanity run passed with no failures: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1799/
To explicitly answer your question https://github.com/AdoptOpenJDK/TSC/issues/132#issuecomment-575528636 @gdams, yes, I am +1 to shipping 2ndary platforms for jdk11 hotspot.
@smlambert thanks I'll ship them now!
jdk-11.0.6+10_openj9-0.18.0 summary so far:
Pipelines Ready for Publish, no blocking failures:
Waiting:
@tsc jdk-11.0.6+10_openj9-0.18.0 on all platforms except AIX, is ready for "Publishing", all tests good with no-blockers. Permission to publish please?
+1
+2
All platforms bar AIX Published. AIX build started: https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/715/
jdk-11.0.6+10_openj9-0.18.0 AIX test failures:
java/util/logging/LogManagerAppContextDeadlock.java.LogManagerAppContextDeadlock | 11 sec | 1 java/lang/ProcessBuilder/Basic.java#id0.Basic_id0 | 43 sec | 2 java/lang/invoke/condy/CondyNestedResolutionTest.java.CondyNestedResolutionTest | 12 sec | 2 java/nio/channels/AsyncCloseAndInterrupt.java.AsyncCloseAndInterrupt | 16 min | 2 java/util/concurrent/locks/Lock/TimedAcquireLeak.java.TimedAcquireLeak
Re-testing... Consistently fail...
@smlambert Are you ok to published with the above failures on AIX?
This is same as for the jdk13 builds (my comment https://github.com/AdoptOpenJDK/TSC/issues/134#issuecomment-576399648), so we can proceed to publish AIX given the extra sanity.functional tests against openj9 for jdk_lang functionality.
Thanks @smlambert i'll go ahead and publish
jdk-11.0.6+10_openj9-0.18.0 all platforms now published
closing as the release is complete
Java Version: jdk-11.0.6+10
JVM:
@AdoptOpenJDK/tsc please can somebody +1 this request?