adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
131 stars 314 forks source link

Jan 2024 CPU - Temurin JDK21 release triage #4983

Closed smlambert closed 9 months ago

smlambert commented 10 months ago

Release Summary Report for release-openjdk21-pipeline

Report generated at: Wed, 17 Jan 2024 02:03:02 GMT

TRSS Build and TRSS Grid View Jenkins Build URL https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/19/ Started by upstream project "build-scripts/utils/releaseTrigger_jdk21u" build number 1682 at 1/16/2024, 6:42:10 PM


✔️ results in these Tables means the activity has successfully completed. ⏳ results means that we are actively working on closing off the runs needed for this version, platform, binaryType. ⛔ means there is no build planned for that version/platform combination. ⏸️ means activity not yet started.

Platform Status Results
x86-64_mac ✔️ comment
x86-64_linux ✔️ comment- raise issue against sanity.openjdk failures that occur on some machines but not others https://github.com/adoptium/aqa-tests/issues/5012
x86-64_windows ✔️ comment
aarch64_linux ✔️ comment
aarch64_mac ✔️ all pass
x86-64_alpine-linux ✔️ comment with post-release actions https://github.com/adoptium/temurin-build/issues/3623
ppc64le_linux ✔️ build failed, will need a rerun comment
s390x_linux ✔️ comment
ppc64_aix ✔️ comment, with post-release actions https://github.com/adoptium/aqa-tests/issues/5010
aarch64_alpine-Linux ✔️ comment
smlambert commented 10 months ago

aarch64_linux

aarch64_Jdk21

jdk21_aarch64Linux_AQAvitTaps.zip

waiting on sanity.openjdk and extended.openjdk results

Sophia: sanity.openjdk https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_aarch64_linux/219/ rerun failed target https://ci.adoptium.net/job/Grinder/8567/ 3 failures java/util/StringJoiner/MergeTest.java.MergeTest java/util/StringJoiner/StringJoinerTest.java.StringJoinerTest java/util/zip/InflaterInputStream/UnexpectedEndOfZlibStream.java.UnexpectedEndOfZlibStream

extended.openjdk https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux/62/console : https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux_testList_1/58/ - abort https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux_testList_2/58/ - failed https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux_testList_0/59/ - abort

_[SXA 22/Jan: Also trying that list at extended.openjdk#64 as a test of a container on a new OSUOSL aarch64 machine] [SXA 22/Jan: Also having looked at the Grinder 3994 above it's running on the slower Linaro machine and with concurrency:1 and it's getting a lot of timouts so I've kicked off another run there as Grinder#4003 in the hope it'll get scheudled on the faster machine and run to completion - trying all approaches 😺 ]_ Grinder#4003 - passed Grinder_20240122212240_jdk21_aaarch64_linux.tap.txt

all other top-level targets passing

smlambert commented 10 months ago

x86-64_mac

jdk21_x64Mac_AQAvitTaps.zip

All top-level targets pass

smlambert commented 10 months ago

x86-64_linux
jdk21_x64Linux_AQAvitTaps.zip

Screenshot 2024-01-18 at 8 30 00 AM

TAPverification shows: sanity.openjdk - 5 targets fail, jdk_lang_0, jdk_lang_1, jdk_util_0, jdk_util_1 and jdk_foreign_0 - rerun in Grinder/8539 fail, rerun in Grinder/8555 on test-docker-ubuntu2204-x64-1 - no UPSTREAM_BUILD_JOB_NUMBER passed through so Grinder failed (see https://github.com/adoptium/aqa-tests/issues/4972), rerun in Grinder/8557 passes Grinder_20240118175328_jdk21_x64Linux.tap.txt

ACTION: check if issue exists to capture the failing testcases from jdk_lang, jdk_util and jdk foreign that can pass on test-docker-ubuntu2204-x64-1 but fail on test-docker-ubuntu2204-x64-3

waiting on extended.openjdk results, some failures, some targets passing on rerun on test-equinix_esxi-ubuntu2204-x64-1

all other top-level test targets passing

smlambert commented 10 months ago

x86-64_windows

jdk21_Windows_AQAvitTaps.zip

waiting on sanity.system and extended.openjdk results all other top-level targets pass

Sophia: extended.openjdk results https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows/36/ rerun build failed with 9 failures https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows_rerun/1/#showFailuresLink rerun grinder : https://ci.adoptium.net/job/Grinder/8568/ 4 tests failed tools/jpackage/share/PerUserCfgTest.java.PerUserCfgTest tools/jpackage/share/ServiceTest.java.ServiceTest tools/jpackage/windows/Win8282351Test.java.Win8282351Test tools/jpackage/windows/WinLongVersionTest.java#id1.WinLongVersionTest_id1 rerun on test-azure-win2012r2-x64-3 https://ci.adoptium.net/job/Grinder/8570/ There are no nodes with the label ‘[test-azure-win2012r2-x64-3](https://ci.adoptium.net/label/test-azure-win2012r2-x64-3/)’ - rerun in Grinder/8579 and in Grinder/8581 on test-azure-win2016-x64-1

smlambert commented 10 months ago

x86-64_alpine-linux

Screen Shot 2024-01-19 at 11 43 28 AM

jdk21_alpineLinux_AQAvitTaps.zip

  - testList_2 failures rerun in [Grinder/8617](https://ci.adoptium.net/job/Grinder/8617) - all pass except `build/AbsPathsInImage.java` which fails with `Execution failed: 'main' threw exception: java.lang.OutOfMemoryError: Java heap space` - rerun AbsPathsInImage.java in [TCGrinder/3997](https://ci.eclipse.org/temurin-compliance/job/Grinder/3997/) - passes 
[Grinder_20240122182411_jdk21alpine.tap.txt](https://github.com/adoptium/aqa-tests/files/14015402/Grinder_20240122182411_jdk21alpine.tap.txt)

Failures indicate that the upstream build job did not produce all of the native test libs expected:

could not load native lib: java.lang.UnsatisfiedLinkError: no TestFloatJNIArgs in java.library.path: /home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native
Error occurred during initialization of VM
Could not find agent library TagMapTest on the library path, with error: Error loading shared library libTagMapTest.so: No such file or directory

SL/Jan23 action: To look into testimage to see what it contains (to try and understand the native test failures above

/alpine/jdk-21.0.2+13-test-image/hotspot/jtreg/native # nm *.so | grep TestFloat
libTestFloatJNIArgs.so:
00000000000011f0 T Java_compiler_floatingpoint_TestFloatJNIArgs_add10floats
0000000000001280 T Java_compiler_floatingpoint_TestFloatJNIArgs_add15doubles
00000000000011a0 T Java_compiler_floatingpoint_TestFloatJNIArgs_add15floats
0000000000001220 T Java_compiler_floatingpoint_TestFloatJNIArgs_addFloatsInts
libTestFloatSyncJNIArgs.so:
0000000000001260 T Java_compiler_floatingpoint_TestFloatSyncJNIArgs_combine15doubles
00000000000011a0 T Java_compiler_floatingpoint_TestFloatSyncJNIArgs_combine15floats
/alpine/jdk-21.0.2+13-test-image/hotspot/jtreg/native # 
/alpine/jdk-21.0.2+13-test-image/hotspot/jtreg/native # ldd libTestFloatJNIArgs.so 
    /lib/ld-musl-x86_64.so.1 (0x7f4cca63c000)
/alpine/jdk-21.0.2+13-test-image/hotspot/jtreg/native # ldd libTestFloatSyncJNIArgs.so
    /lib/ld-musl-x86_64.so.1 (0x7f1ce9249000)

I believe that the shared libaries are not in the expected/right location for alpine-linux for some reason, which causes 3 testcases to fail. Recommending to not block the release with these 3 testcases and to exclude, but will raise a build issue with the goal to fix so we can reinclude these tests when ready.

all other top-level targets pass

smlambert commented 10 months ago

ppc64le_linux rerun in https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/21/ - Grid view

Release Summary Report for release-openjdk21-pipeline

Report generated at: Wed, 17 Jan 2024 19:34:22 GMT

TRSS Build and TRSS Grid View Jenkins Build URL https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/21/ Started by upstream project "build-scripts/utils/releaseTrigger_jdk21u" build number 1682 at 1/17/2024, 9:37:43 AM


Screenshot 2024-01-19 at 10 18 57 PM
list AQAvit TAP files in /Users/shelleylambert/Downloads/AQAvitTaps
============================
1. List sanity.openjdk ...
Test_openjdk21_hs_sanity.openjdk_ppc64le_linux.tap
2. List extended.openjdk ...
Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_0.tap
Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap
Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_2.tap
3. List sanity.functional ...
Test_openjdk21_hs_sanity.functional_ppc64le_linux.tap
4. List extended.functional ...
Test_openjdk21_hs_extended.functional_ppc64le_linux.tap
5. List special.functional ...
Test_openjdk21_hs_special.functional_ppc64le_linux.tap
6. List sanity.system ...
Test_openjdk21_hs_sanity.system_ppc64le_linux.tap
7. List extended.system ...
Test_openjdk21_hs_extended.system_ppc64le_linux.tap
8. List sanity.perf ...
Test_openjdk21_hs_sanity.perf_ppc64le_linux.tap
9. List extended.perf ...
Test_openjdk21_hs_extended.perf_ppc64le_linux.tap
============================
List failed tests in /Users/shelleylambert/Downloads/AQAvitTaps
============================
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_0.tap:not ok 12 - jdk_tools_1
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 3 - jvm_compiler_0
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 13 - jdk_security3_1
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 19 - jdk_management_0
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 20 - jdk_management_1
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 25 - jdk_jfr_1
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 38 - jdk_vector_0
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_1.tap:not ok 39 - jdk_vector_1
./Test_openjdk21_hs_extended.openjdk_ppc64le_linux_testList_2.tap:not ok 26 - jdk_jdi_1
./Test_openjdk21_hs_sanity.openjdk_ppc64le_linux.tap:not ok 1 - jdk_lang_0
./Test_openjdk21_hs_sanity.openjdk_ppc64le_linux.tap:not ok 2 - jdk_lang_1
./Test_openjdk21_hs_sanity.openjdk_ppc64le_linux.tap:not ok 22 - jdk_util_0
./Test_openjdk21_hs_sanity.openjdk_ppc64le_linux.tap:not ok 23 - jdk_util_1

sanity.openjdk

extended.openjdk

smlambert commented 10 months ago

s390x_linux

Screenshot 2024-01-19 at 10 20 10 PM

jdk21_s390x_AQAvitTaps.zip

extended.openjdk failed test targets:

all other top-level targets pass

smlambert commented 10 months ago

ppc64_aix

Screenshot 2024-01-19 at 9 40 01 PM

jdk21_aix_AQAvitTaps.zip

============================
list AQAvit TAP files in /Users/shelleylambert/Downloads/AQAvitTaps
============================
1. List sanity.openjdk ...
Test_openjdk21_hs_sanity.openjdk_ppc64_aix.tap
2. List extended.openjdk ...
*extended.openjdk*.tap
3. List sanity.functional ...
Test_openjdk21_hs_sanity.functional_ppc64_aix.tap
4. List extended.functional ...
Test_openjdk21_hs_extended.functional_ppc64_aix.tap
5. List special.functional ...
Test_openjdk21_hs_special.functional_ppc64_aix.tap
6. List sanity.system ...
Test_openjdk21_hs_sanity.system_ppc64_aix.tap
7. List extended.system ...
Test_openjdk21_hs_extended.system_ppc64_aix.tap
8. List sanity.perf ...
Test_openjdk21_hs_sanity.perf_ppc64_aix.tap
9. List extended.perf ...
Test_openjdk21_hs_extended.perf_ppc64_aix.tap
============================
List failed tests in /Users/shelleylambert/Downloads/AQAvitTaps
============================
./Test_openjdk21_hs_extended.perf_ppc64_aix.tap:not ok 20 - renaissance-philosophers_0

Recommend to exclude renaissance-philosophers for AIX, as we do not have sufficient memory to run it on test machines

#
# There is insufficient memory for the Java Runtime Environment to continue.
[thread 8690 also had an error]# Native memory allocation 
(malloc) failed to allocate 336 bytes for AllocateHeap[thread 11517 also had an error]

Actions:

smlambert commented 9 months ago

aarch64_alpine-Linux

Screen Shot 2024-01-19 at 9 31 48 AM

Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux ❌ ABORTED ❌

Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_0 ❌ ABORTED ❌ jdk_beans_1 => deep history 0/2 passed | possible issues jdk_security3_0 => deep history 0/1 passed | possible issues jdk_net_0 => deep history 0/1 passed | possible issues jdk_time_1 => deep history 0/1 passed | possible issues jdk_jmx_0 => deep history 8/9 passed | possible issues jdk_jfr_1 => deep history 0/1 passed | possible issues

Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_1 ❌ ABORTED ❌ jvm_compiler_0 => deep history 0/1 passed | possible issues

Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_2 ❌ ABORTED ❌ jdk_io_1 => deep history 0/1 passed | possible issues jdk_other_1 => deep history 0/1 passed | possible issues jdk_net_1 => deep history 8/9 passed | possible issues jdk_nio_1 => deep history 8/9 passed | possible issues jdk_text_0 => deep history 8/9 passed | possible issues jdk_management_0 => deep history 8/9 passed | possible issues

Test_openjdk21_hs_sanity.openjdk_aarch64_alpine-linux ⚠️ UNSTABLE ⚠️ Rerun failed

java -version ``` openjdk version "21.0.2" 2024-01-16 LTS OpenJDK Runtime Environment Temurin-21.0.2+13 (build 21.0.2+13-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (build 21.0.2+13-LTS, mixed mode, sharing) ```

jdk_util_0 => deep history 11/13 passed | possible issues

Test_openjdk21_hs_sanity.openjdk_aarch64_alpine-linux_rerun ⚠️ UNSTABLE ⚠️ Rerun failed

java -version ``` openjdk version "21.0.2" 2024-01-16 LTS OpenJDK Runtime Environment Temurin-21.0.2+13 (build 21.0.2+13-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (build 21.0.2+13-LTS, mixed mode, sharing) ```
Failures in the rerun test build [jdk_util_0](https://trss.adoptium.net/output/test?id=65a8571196a988006d35d4b2) [jdk_util_0](https://trss.adoptium.net/output/test?id=65a8571196a988006d35d4b5) [jdk_util_0](https://trss.adoptium.net/output/test?id=65a8571196a988006d35d4b3)

Sophia: sanity: 2 tests failed in sanity https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_aarch64_alpine-linux/18/ rerun build run the target and got other tests failing, the failed two testcases passed. rerun failed testcases grinder https://ci.adoptium.net/job/Grinder/8642/ - passed Grinder_20240122155604_jdk21_aarch64_alpine.tap.txt

extended: Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_0 ❌ ABORTED ❌ - 15 test targets 5 passed targets: hotspot_serviceability_jvmti_0_PASSED, jdk_io_0_PASSED,jdk_other_0_PASSED,jdk_nio_0_PASSED, jdk_jfr_0_PASSED

5 failed targets: jdk_bean_1 ( TEST: java/beans/Introspector/MethodOrderException.java), jdk_net_0_FAILED (),jdk_security3_0_FAILED,jdk_time_1_FAILED, jdk_jmx_0_FAILED(javax/management/remote/mandatory/loading/MissingClassTest.java),

5 aborted: jdk_jfr_1, jdk_jdi_0,jdk_imageio_0, jdk_native_sanity_1,jdk_vector_1 rerun failed or aborted test target https://ci.adoptium.net/job/Grinder/8643/ https://ci.eclipse.org/temurin-compliance/view/Grinder/job/Grinder/4009/ Grinder_20240123162538_jdk21_alpine_linux.tap.txt

6 testcases failed. rerun https://ci.eclipse.org/temurin-compliance/job/Grinder/4020 Grinder_20240124043808_jdk21_alpine_linux.tap.txt

testlist_1: rebuild https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_1/12/ https://ci.eclipse.org/temurin-compliance/view/Grinder/job/Grinder/4008/ - passed ( job shows unstable for other reason) Grinder_20240123152401_jdk21_aarch64_alpine.tap.txt

testlist_2: TESTLIST=jvm_native_sanity_1,jdk_net_1,jdk_instrument_0,jdk_nio_1,jdk_build_0,jdk_build_1,jdk_imageio_1,jdk_text_0,jdk_vector_0,jvm_native_sanity_0,jdk_io_1,jdk_management_0,jdk_other_1,jdk_rmi_0,jdk_rmi_1 rebuild https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_alpine-linux_testList_2/11/ - 15 testcases failing rerun https://ci.adoptium.net/job/Grinder/8654/ - failed. rerun in eclipse temurin https://ci.eclipse.org/temurin-compliance/view/Grinder/job/Grinder/4005/ Grinder_20240122222030_jdk21_alpine_linux.tap.txt

rerun 10 failed testcases https://ci.eclipse.org/temurin-compliance/view/Grinder/job/Grinder/4019/ - passed

Grinder_20240124043145_jdk21_alpine_linux.tap.txt