adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
132 stars 312 forks source link

Pre-release triage of weekly test runs #2754

Closed smlambert closed 2 years ago

smlambert commented 3 years ago

This report represents this view:

Screen Shot 2021-07-18 at 11 11 43 AM

Notable:


TRSS link Build URL https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/1709/ Started by upstream project "build-scripts/weekly-openjdk8-pipeline" build number 30

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_0 hotspot_jre_0 => deep history 0/2 passed | possible issues jdk_jmx_0 => deep history 0/3 passed | possible issues jdk_jfr_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_sparcv9_solaris ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.perf_x86-32_windows renaissance-als_0 => deep history 0/5 passed | possible issues renaissance-chi-square_0 => deep history 0/5 passed | possible issues renaissance-dec-tree_0 => deep history 0/5 passed | possible issues renaissance-gauss-mix_0 => deep history 0/5 passed | possible issues renaissance-log-regression_0 => deep history 0/5 passed | possible issues renaissance-movie-lens_0 => deep history 0/5 passed | possible issues

Test_openjdk8_hs_sanity.perf_sparcv9_solaris renaissance-scala-kmeans_0 => deep history 0/6 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows_testList_0 hotspot_jre_0 => deep history 0/2 passed | possible issues jdk_jmx_0 => deep history 0/3 passed | possible issues jdk_jfr_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_sanity.system_arm_linux TestJlmRemoteClassAuth_2 => deep history 12/13 passed | possible issues TestJlmRemoteClassNoAuth_2 => deep history 8/13 passed | possible issues TestJlmRemoteMemoryAuth_2 => deep history 10/13 passed | possible issues TestJlmRemoteMemoryNoAuth_2 => deep history 7/13 passed | possible issues TestJlmRemoteThreadNoAuth_2 => deep history 9/13 passed | possible issues

Test_openjdk8_hs_sanity.perf_x86-64_solaris renaissance-scala-kmeans_0 => deep history 0/7 passed | possible issues

[Test_openjdk8_hs_sanity.openjdk_arm_linux] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_arm_linux/247/) Triage Done jdk_util_2 => deep history 1/13 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows_testList_1 jdk_jmx_1 => deep history 0/3 passed | possible issues jdk_jfr_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_linux ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_2 hotspot_jre_1 => deep history 0/4 passed | possible issues jdk_management_0 => deep history 0/3 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_1 jdk_management_1 => deep history 0/3 passed | possible issues jdk_jmx_1 => deep history 0/3 passed | possible issues jdk_jfr_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_1 jdk_jfr_2 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_arm_linux ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_1 hotspot_jre_2 => deep history 0/2 passed | possible issues jdk_net_2 => deep history 1/4 passed | possible issues jdk_jfr_2 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_solaris ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_2 jdk_beans_2 => deep history 1/2 passed | possible issues jdk_nio_2 => deep history 1/4 passed | possible issues jdk_tools_2 => deep history 0/4 passed | possible issues jdk_imageio_2 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_solaris_testList_0 hotspot_jre_0 => deep history 0/2 passed | possible issues hotspot_jre_1 => deep history 0/2 passed | possible issues hotspot_jre_2 => deep history 0/2 passed | possible issues jdk_net_0 => deep history 0/2 passed | possible issues jdk_net_1 => deep history 0/2 passed | possible issues jdk_net_2 => deep history 0/2 passed | possible issues jdk_nio_1 => deep history 1/2 passed | possible issues jdk_security2_0 => deep history 0/2 passed | possible issues jdk_security2_1 => deep history 0/2 passed | possible issues jdk_security2_2 => deep history 0/2 passed | possible issues jdk_jmx_0 => deep history 0/2 passed | possible issues jdk_jmx_1 => deep history 0/2 passed | possible issues jdk_jmx_2 => deep history 0/2 passed | possible issues jdk_rmi_0 => deep history 0/2 passed | possible issues jdk_rmi_1 => deep history 0/2 passed | possible issues jdk_rmi_2 => deep history 0/2 passed | possible issues jdk_tools_0 => deep history 0/2 passed | possible issues jdk_tools_1 => deep history 0/2 passed | possible issues jdk_tools_2 => deep history 0/2 passed | possible issues jdk_jfr_0 => deep history 0/2 passed | possible issues jdk_jfr_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_2 hotspot_jre_1 => deep history 0/3 passed | possible issues jdk_beans_1 => deep history 1/2 passed | possible issues jdk_imageio_0 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64le_linux ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_x86-64_mac_testList_2 jdk_instrument_0 => deep history 0/3 passed | possible issues

Test_openjdk8_hs_extended.openjdk_aarch64_linux hotspot_jre_0 => deep history 0/4 passed | possible issues hotspot_jre_1 => deep history 1/4 passed | possible issues jdk_management_0 => deep history 0/4 passed | possible issues jdk_management_1 => deep history 0/4 passed | possible issues jdk_jmx_0 => deep history 0/4 passed | possible issues jdk_jmx_1 => deep history 0/4 passed | possible issues jdk_jfr_0 => deep history 0/4 passed | possible issues jdk_jfr_1 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.perf_arm_linux dacapo-xalan_0 => deep history 1/4 passed | possible issues renaissance-als_0 => deep history 0/4 passed | possible issues renaissance-chi-square_0 => deep history 0/4 passed | possible issues renaissance-dec-tree_0 => deep history 0/4 passed | possible issues renaissance-gauss-mix_0 => deep history 0/4 passed | possible issues renaissance-log-regression_0 => deep history 0/4 passed | possible issues renaissance-movie-lens_0 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_1 jdk_security4_1 => deep history 0/3 passed | possible issues jdk_jmx_1 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_windows ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_2 hotspot_jre_1 => deep history 0/3 passed | possible issues jdk_management_0 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_sanity.openjdk_ppc64_aix jdk_util_0 => deep history 9/13 passed | possible issues jdk_util_1 => deep history 9/13 passed | possible issues

Test_openjdk8_hs_extended.openjdk_arm_linux_testList_0 jdk_management_2 => deep history 0/4 passed | possible issues jdk_jmx_2 => deep history 0/4 passed | possible issues

Test_openjdk8_hs_extended.openjdk_ppc64_aix_testList_0 hotspot_jre_0 => deep history 0/2 passed | possible issues jdk_security4_0 => deep history 0/2 passed | possible issues jdk_jmx_0 => deep history 0/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-32_windows ⚠️ Test Job Failed ⚠️

Test_openjdk8_hs_sanity.system_x86-64_solaris TestJlmRemoteMemoryAuth_0 => deep history 5/7 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_mac_testList_1 jdk_nio_1 => deep history 1/2 passed | possible issues

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0 Triage done hotspot_jre_0 => deep history 0/1 passed | possible issues jdk_jmx_0 => deep history 0/2 passed | possible issues jdk_jfr_1 => deep history 0/1 passed | possible issues

Test_openjdk8_hs_extended.openjdk_sparcv9_solaris_testList_0 hotspot_jre_0 => deep history 0/2 passed | possible issues jdk_security2_0 => deep history 0/2 passed | possible issues jdk_jmx_0 => deep history 0/2 passed | possible issues jdk_jfr_0 => deep history 0/2 passed | possible issues jdk_imageio_0 => deep history 0/2 passed | possible issues

[Test_openjdk8_hs_sanity.openjdk_x86-64_windows] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/509/) Triage Done ⚠️ Test Job Failed ⚠️

[Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_0] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_0/10/) Triage Done jdk_jmx_2 => deep history 0/4 passed | possible issues

[Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1] (https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1/9/) Triage Done jdk_management_1 => deep history 1/2 passed | possible issues jdk_jmx_1 => deep history 0/2 passed | possible issues jdk_jfr_0 => deep history 0/1 passed | possible issues

smlambert commented 3 years ago

Note: jdk_jfr target seems to be failing on many platforms with "needs -nativepath" error (jdk/jfr/event/sampling/TestNative.java testcase in that target that requires it), perhaps need to do a similar thing that has been done for jdk_foreign_native target? or simply exclude jdk/jfr/event/sampling/TestNative.java in problem list if its failing everywhere.

FYI @sophia-guo

smlambert commented 3 years ago

jdk_jmx target failing with:

[2021-07-17T14:16:43.687Z] STDERR:
[2021-07-17T14:16:43.687Z] /home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_ppc64le_linux_testList_0/aqa-tests/openjdk/openjdk-jdk/jdk/test/javax/management/remote/mandatory/connection/RMIConnector_NPETest.java:64: error: method shutdown in class RMID cannot be applied to given types;
[2021-07-17T14:16:43.687Z]             rmid.shutdown(rmidPort);
[2021-07-17T14:16:43.687Z]                 ^
[2021-07-17T14:16:43.687Z]   required: no arguments
[2021-07-17T14:16:43.687Z]   found: int
[2021-07-17T14:16:43.687Z]   reason: actual and formal argument lists differ in length
[2021-07-17T14:16:43.687Z] 1 error
[2021-07-17T14:16:43.687Z] 

This is https://bugs.openjdk.java.net/browse/JDK-8066588, which appears resolved, check why we do not have the fix in our mirror.

sophia-guo commented 3 years ago

We don't build test images for jdk8 so I will exclude it for 8.

Curious thing is the test isn't excluded for jdk11 however it doesn't run in jdk11. https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_hs_extended.openjdk_x86-64_linux_testList_2/7/testReport/

Tried with jdk_custom with jdk11 and looks like it failed. https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/152/

Turns out for jdk11 the test was excluded in upstream https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/test/jdk/ProblemList.txt#L959

sophia-guo commented 3 years ago

Re https://github.com/adoptium/aqa-tests/issues/2754#issuecomment-882562339

Yes, looks like it was backported to jdk8

JDK-8263145 | 8u301 | Yoshiki Sato | P3 | Resolved | Fixed | b01 https://bugs.openjdk.java.net/browse/JDK-8263145

Though I don't quite understand the backport process. Check the HG https://hg.openjdk.java.net/jdk8u/jdk8u/ and github https://github.com/openjdk/jdk8u/blob/master/jdk/test/javax/management/remote/mandatory/connection/RMIConnector_NPETest.java the change is not there.

Exclude the test as the backport should be propagated correctly in upstream repo.

smlambert commented 3 years ago

Issues on x86_solaris:

Definitions:

: [:] ``` - com/sun/crypto/provider/Mac/MacClone.java fails with `java.lang.CloneNotSupportedException: SHA-256`
sophia-guo commented 3 years ago

Issues for perf on sparcv9_solaris: https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.perf_x86-64_solaris/10/console

Test renaissance-scala-kmeans_0 failed without any error message. Suspect the way the test is triggered may have some issue. Looking at the passing test on linux, there is message Warning!!! Affinity is NOT set. Affinity tool may NOT be installed/supported. Believe that something wrong with dynamically set CPU affinities added in https://github.com/adoptium/aqa-tests/pull/1917

08:32:18  Warning!!! Affinity is NOT set. Affinity tool may NOT be installed/supported.
08:32:18  Running EXEC_CMD_WITH_AFFINITY= /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/openjdkbinary/j2sdk-image/bin/java -jar /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/aqa-tests/TKG/../../jvmtest/perf/renaissance/renaissance.jar --json /home/jenkins/workspace/Test_openjdk8_hs_sanity.perf_x86-64_linux/aqa-tests/TKG/../TKG/output_16265245182190/renaissance-scala-kmeans_0/scala-kmeans.json scala-kmeans

Will remove dynamically setting CPU affinities and need someone to look at the issues in run_with_affinity.sh

sophia-guo commented 3 years ago

jdk_util test java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java intermittently failed on arm build Test_openjdk8_hs_sanity.openjdk_arm_linux It is also machine related. Suspect related with arm memory size? And this is not a regression failure, which should not block the release, will exclude it? @smlambert

1/10 failed build-equinix-ubuntu1604-armv7l-1 : https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/162/tapResults/ 7/10 failed test-aahlenst-armv7-raspbian-pi4-1 : https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/163/tapResults/

09:19:54  test org.openjdk.tests.java.util.stream.MapOpTest.testOps("ArrayList.asList:pseudorandom", java.util.stream.TestData$AbstractTestData$RefTestData@98fbd2): failure
09:19:54  java.lang.AssertionError: 1 failure(s) for test data: RefTestData[ArrayList.asList:pseudorandom]
09:19:54  1: java.lang.Error: PAR_STREAM_FOR_EACH_CLEAR_SIZED: java.lang.ArrayIndexOutOfBoundsException
09:19:54  java.lang.ArrayIndexOutOfBoundsException
smlambert commented 3 years ago

re: https://github.com/adoptium/aqa-tests/issues/2754#issuecomment-883704362 yes, thanks @sophia-guo

sophia-guo commented 3 years ago

Test_openjdk8_hs_sanity.openjdk_x86-64_windows failed with fatal: unable to access 'https://github.com/eclipse-openj9/openj9.git/'. Either it's a network issue or a machine issue? Re-ran: will see if there are specific test failures

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/510/

09:38:22  get functional test material...
09:38:22  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
09:38:45  fatal: unable to access 'https://github.com/eclipse-openj9/openj9.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 

Test_openjdk8_hs_sanity.openjdk_x86-64_windows : rerun green

sophia-guo commented 3 years ago

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_1 jdk_management_1 => deep history 1/2 passed | possible issues

Test com/sun/management/OperatingSystemMXBean/TestTotalSwap.java fails when run in container exclude for platform run in container. Suppose only for linux we run tests in docker.

sophia-guo commented 3 years ago

Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0

hotspot_jre_0 => deep history 0/1 passed | possible issues:

Following test failures were reported in https://bugs.openjdk.java.net/browse/JDK-8180723?focusedCommentId=14086592&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14086592 . Closed as cannot reproduce, which should be reopen.

compiler/rtm/locking/TestUseRTMAfterLockInflation.java.TestUseRTMAfterLockInflation compiler/rtm/locking/TestUseRTMForInflatedLocks.java.TestUseRTMForInflatedLocks compiler/rtm/locking/TestUseRTMForStackLocks.java.TestUseRTMForStackLocks compiler/rtm/method_options/TestUseRTMLockElidingOption.java.TestUseRTMLockElidingOption

jdk/jfr/event/compiler/TestCompilerCompile.java.TestCompilerCompile : timeout. Running in the docker. Rerun on nondocker, passed: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/171/

Program `/home/jenkins/workspace/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960351ms)

jdk/jfr/jcmd/TestJcmdDumpPathToGCRoots.java error message: "Execution failed: `main' threw exception: java.lang.Exception: Expected chains but found none " issue opened https://github.com/adoptium/aqa-tests/issues/2766

sophia-guo commented 3 years ago

Test_openjdk8_hs_extended.openjdk_x86-32_windows Failed tests specific to x86-32:

jdk/jfr/event/compiler/TestCompilerPhase.java.TestCompilerPhase

 Stack TraceExecution failed: `main' threw exception: java.lang.RuntimeException: No events: expected false, was true  Standard Output Standard Error 2 sec 2

jdk/jfr/event/oldobject/TestObjectDescription.java.TestObjectDescription 

Stack TraceProgram `E:\workspace\Test_openjdk8_hs_extended.openjdk_x86-32_windows_testList_1\openjdkbinary\j2sdk-image\bin\java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960006ms).  Standard Output Standard Error

karianna commented 3 years ago

Recently for jdk8u312-b07 - the same security failures happened https://trss.adoptium.net/output/test?id=616ff8ccc6182d02178e14af

SehrishHussain commented 3 years ago

Another test fail related to jdk-jfr for openjdk18, ppc64le_linux: jdk/jfr/jmx/TestGetRecordings.java

Jenkins URL: https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.openjdk_ppc64le_linux_testList_0/7/#showFailuresLink TRSS Link: https://trss.adoptium.net/allTestsInfo?buildId=616d7c2bc6182d021780004e&limit=5&hasChildren=true

smlambert commented 2 years ago

Closing this as both 'completed' and stale.

zzambers commented 8 months ago

We don't build test images for jdk8 so I will exclude it for 8.

Jdk8 actually does not have test images. Test was fixed by JDK-8301143: jdk/jfr/event/sampling/TestNative.java

however I have found that it still fails on MacOS in grinder:

...
14:57:40  + OS=Darwin
14:57:40  + '[' Darwin '!=' Linux ']'
14:57:40  + '[' Darwin '!=' Darwin ']'
14:57:40  + '[' x = x ']'
14:57:40  +++ which gcc
14:57:40  ++ readlink -f /usr/bin/gcc
14:57:40  readlink: illegal option -- f
14:57:40  usage: readlink [-n] [file ...]
14:57:40  + TESTGCC=
...