search

adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
124 stars 303 forks source link

Triage for Amazon Linux 2023 ( Docker static) #5300

Closed sophia-guo closed 3 weeks ago

sophia-guo commented 1 month ago

https://github.com/adoptium/infrastructure/issues/3041#issuecomment-2102784806

https://github.com/adoptium/infrastructure/issues/3041#issuecomment-2102785607

sophia-guo commented 1 month ago

Aarch64 has one more test failure - https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_aarch64_linux/82/testReport/junit/jdk_jfr_jcmd_TestJcmdDump/java/TestJcmdDump/ , which happened to other platforms before.

Other than that those two platforms have exactly same failures. Will triage x64.

sophia-guo commented 1 month ago

sanity

java/lang/ProcessHandle/InfoTest.java.InfoTest java/lang/ProcessHandle/InfoTest.java.InfoTest java/lang/ProcessBuilder/JspawnhelperWarnings.java.JspawnhelperWarnings - not in jdk-21.0.3-ga java/lang/ProcessBuilder/JspawnhelperWarnings.java.JspawnhelperWarnings java/lang/String/CompactString/NegativeSize.java.NegativeSize - not in jdk-21.0.3-ga java/lang/String/CompactString/NegativeSize.java.NegativeSize java/lang/Thread/virtual/JfrEvents.java.JfrEvents java/lang/Thread/virtual/JfrEvents.java.JfrEvents java/lang/Thread/virtual/ThreadAPI.java#default.ThreadAPI_default java/lang/Thread/virtual/ThreadAPI.java#default.ThreadAPI_default java/lang/Thread/virtual/TracePinnedThreads.java.TracePinnedThreads java/lang/Thread/virtual/TracePinnedThreads.java.TracePinnedThreads java/security/Security/ConfigFileTest.java.ConfigFileTest java/security/Security/ConfigFileTest.java.ConfigFileTest java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0.GetStackTraceALotWhenPinned_id0 - not in jdk-21.0.3-ga java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0.GetStackTraceALotWhenPinned_id0

Latest run passed on test-ibmcloud-rhel7-x64-1 https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/197/

Rerun on other docker static test-docker-debian12-x64-4 https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/200/ - same 16 failures

sophia-guo commented 1 month ago

extended 32 failures, same test failures as the run on https://ci.adoptium.net/computer/test-docker-ubuntu2404-x64-1/

https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/53/ on test-docker-amazon2023-x64-1

https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/52/ on https://ci.adoptium.net/computer/test-docker-ubuntu2404-x64-1/

sophia-guo commented 1 month ago

@sxa According to jenkins builds mentioned above- running against jdk21

So Temurin has same validation on Amazon (docker static) as on other docker static. Let me know if any further validation needed.

sxa commented 1 month ago

The Ubuntu 24.04 machine is also new (and will also require more triage of the results on there - there are some failures in those runs which I hadn't seen before), possibly also true for the Debian 12 one which I think is also new, so they may not be the best ones to compare with.

Are those 16 + 32 failures expected and seen on the GA release triage last month? I didn't think I'd seen that many failures in the suite when I was running it recently on builds on other machines.

EDIT: From

Latest run passed on test-ibmcloud-rhel7-x64-1 https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/197/

does this mean that Amazon Linux 2023 is showing failures that do not occur on RHEL7?

sophia-guo commented 1 month ago

Latest run passed on test-ibmcloud-rhel7-x64-1 https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/197/

does this mean that Amazon Linux 2023 is showing failures that do not occur on RHEL7?

test-ibmcloud-rhel7-x64-1 is not a docker static one.

sxa commented 1 month ago

Does that mean we're seeing significantly different results under docker than normal machines with the same tests? I wasn't aware we still had that and that is quite concerning. I tried running a "green" sanity.openjdk from something else I was working on recently on three machines and they were able to run green on the ubuntu 24.04 and ubuntu 20.04 machines but not The Amazon Linux one (although it was just one failure)

I'd be more comfortable if we understood why we're getting failures here before declaring success.

sophia-guo commented 1 month ago

https://ci.adoptium.net/label/ci.role.test&&hw.arch.x86&&sw.os.linux&&!%28centos6%7C%7Crhel6%29/

Which docker static is a good one to compare? We do have some tests failures are related with docker ones, which should not be the issue with the os, arch platform. So I think it's fair to limit the comparison to docker static ones.

smlambert commented 1 month ago

jdk_lang

smlambert commented 1 month ago

jdk_lang

[2024-05-07T16:18:57.326Z] TEST: java/lang/String/CompactString/NegativeSize.java
[2024-05-07T16:18:57.326Z] TEST JDK: /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jdkbinary/j2sdk-image
[2024-05-07T16:18:57.326Z] 
[2024-05-07T16:18:57.326Z] ACTION: build -- Passed. Build successful
[2024-05-07T16:18:57.326Z] REASON: Named class compiled on demand
[2024-05-07T16:18:57.326Z] TIME:   0.034 seconds
[2024-05-07T16:18:57.326Z] messages:
[2024-05-07T16:18:57.326Z] command: build NegativeSize
[2024-05-07T16:18:57.326Z] reason: Named class compiled on demand
[2024-05-07T16:18:57.326Z] started: Tue May 07 16:18:44 UTC 2024
[2024-05-07T16:18:57.326Z] Test directory:
[2024-05-07T16:18:57.326Z]   compile: NegativeSize
[2024-05-07T16:18:57.326Z] finished: Tue May 07 16:18:44 UTC 2024
[2024-05-07T16:18:57.326Z] elapsed time (seconds): 0.034
[2024-05-07T16:18:57.326Z] 
[2024-05-07T16:18:57.326Z] ACTION: compile -- Passed. Compilation successful
[2024-05-07T16:18:57.326Z] REASON: .class file out of date or does not exist
[2024-05-07T16:18:57.326Z] TIME:   0.033 seconds
[2024-05-07T16:18:57.326Z] messages:
[2024-05-07T16:18:57.326Z] command: compile /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString/NegativeSize.java
[2024-05-07T16:18:57.326Z] reason: .class file out of date or does not exist
[2024-05-07T16:18:57.327Z] started: Tue May 07 16:18:44 UTC 2024
[2024-05-07T16:18:57.327Z] Mode: agentvm
[2024-05-07T16:18:57.327Z] Agent id: 3
[2024-05-07T16:18:57.327Z] finished: Tue May 07 16:18:44 UTC 2024
[2024-05-07T16:18:57.327Z] elapsed time (seconds): 0.033
[2024-05-07T16:18:57.327Z] configuration:
[2024-05-07T16:18:57.327Z] Boot Layer (javac runtime environment)
[2024-05-07T16:18:57.327Z]   class path: /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jvmtest/openjdk/jtreg/lib/javatest.jar 
[2024-05-07T16:18:57.327Z]               /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jvmtest/openjdk/jtreg/lib/jtreg.jar 
[2024-05-07T16:18:57.327Z]   patch:      java.base /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/patches/java.base
[2024-05-07T16:18:57.327Z] 
[2024-05-07T16:18:57.327Z] javac compilation environment
[2024-05-07T16:18:57.327Z]   source path: /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString
[2024-05-07T16:18:57.327Z]   class path:  /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString
[2024-05-07T16:18:57.327Z]                /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d
[2024-05-07T16:18:57.327Z] 
[2024-05-07T16:18:57.327Z] rerun:
[2024-05-07T16:18:57.327Z] cd /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/scratch/0 && \
[2024-05-07T16:18:57.327Z] DISPLAY=:0 \
[2024-05-07T16:18:57.327Z] HOME=/home/jenkins \
[2024-05-07T16:18:57.327Z] PATH=/bin:/usr/bin:/usr/sbin \
[2024-05-07T16:18:57.327Z]     /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jdkbinary/j2sdk-image/bin/javac \
[2024-05-07T16:18:57.327Z]         -J-ea \
[2024-05-07T16:18:57.327Z]         -J-esa \
[2024-05-07T16:18:57.327Z]         -J-Xmx512m \
[2024-05-07T16:18:57.327Z]         -J-XX:+UseCompressedOops \
[2024-05-07T16:18:57.327Z]         -J-Dtest.vm.opts='-ea -esa -Xmx512m -XX:+UseCompressedOops' \
[2024-05-07T16:18:57.327Z]         -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-XX:+UseCompressedOops' \
[2024-05-07T16:18:57.327Z]         -J-Dtest.compiler.opts= \
[2024-05-07T16:18:57.327Z]         -J-Dtest.java.opts= \
[2024-05-07T16:18:57.327Z]         -J-Dtest.jdk=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jdkbinary/j2sdk-image \
[2024-05-07T16:18:57.327Z]         -J-Dcompile.jdk=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jdkbinary/j2sdk-image \
[2024-05-07T16:18:57.327Z]         -J-Dtest.timeout.factor=8.0 \
[2024-05-07T16:18:57.327Z]         -J-Dtest.nativepath=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/jdkbinary/openjdk-test-image/jdk/jtreg/native \
[2024-05-07T16:18:57.327Z]         -J-Dtest.root=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk \
[2024-05-07T16:18:57.327Z]         -J-Dtest.name=java/lang/String/CompactString/NegativeSize.java \
[2024-05-07T16:18:57.327Z]         -J-Dtest.file=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString/NegativeSize.java \
[2024-05-07T16:18:57.327Z]         -J-Dtest.src=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString \
[2024-05-07T16:18:57.327Z]         -J-Dtest.src.path=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString \
[2024-05-07T16:18:57.327Z]         -J-Dtest.classes=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d \
[2024-05-07T16:18:57.327Z]         -J-Dtest.class.path=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d \
[2024-05-07T16:18:57.327Z]         -J-Dtest.class.path.prefix=/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d:/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString \
[2024-05-07T16:18:57.327Z]         -d /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d \
[2024-05-07T16:18:57.327Z]         -sourcepath /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString \
[2024-05-07T16:18:57.327Z]         -classpath /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString:/home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/TKG/output_17150982492999/jdk_lang_0/work/classes/1/java/lang/String/CompactString/NegativeSize.d /home/jenkins/workspace/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString/NegativeSize.java
[2024-05-07T16:18:57.327Z] 
[2024-05-07T16:18:57.327Z] ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Failed: Expected OutOfMemoryError
[2024-05-07T16:18:57.327Z] REASON: User specified action: run main/othervm -XX:+CompactStrings -Xmx4g NegativeSize 
[2024-05-07T16:18:57.327Z] TIME:   4.882 seconds
[2024-05-07T16:18:57.327Z] messages:
[2024-05-07T16:18:57.327Z] command: main -XX:+CompactStrings -Xmx4g NegativeSize
[2024-05-07T16:18:57.327Z] reason: User specified action: run main/othervm -XX:+CompactStrings -Xmx4g NegativeSize 
[2024-05-07T16:18:57.327Z] started: Tue May 07 16:18:44 UTC 2024
[2024-05-07T16:18:57.327Z] Mode: othervm [/othervm specified]
[2024-05-07T16:18:57.327Z] finished: Tue May 07 16:18:49 UTC 2024
[2024-05-07T16:18:57.327Z] elapsed time (seconds): 4.882
[2024-05-07T16:18:57.327Z] configuration:
[2024-05-07T16:18:57.327Z] STDOUT:
[2024-05-07T16:18:57.327Z] STDERR:
[2024-05-07T16:18:57.327Z] java.lang.RuntimeException: Failed: Expected OutOfMemoryError
[2024-05-07T16:18:57.327Z]  at NegativeSize.main(NegativeSize.java:70)
[2024-05-07T16:18:57.327Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-05-07T16:18:57.327Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-05-07T16:18:57.327Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2024-05-07T16:18:57.327Z]  at java.base/java.lang.Thread.run(Thread.java:1583)
[2024-05-07T16:18:57.327Z] Caused by: java.lang.NegativeArraySizeException: -1894967266
[2024-05-07T16:18:57.327Z]  at java.base/java.lang.String.<init>(String.java:577)
[2024-05-07T16:18:57.327Z]  at java.base/java.lang.String.<init>(String.java:1425)
[2024-05-07T16:18:57.327Z]  at NegativeSize.main(NegativeSize.java:60)
[2024-05-07T16:18:57.327Z]  ... 4 more
[2024-05-07T16:18:57.327Z] 
[2024-05-07T16:18:57.327Z] JavaTest Message: Test threw exception: java.lang.RuntimeException: Failed: Expected OutOfMemoryError
[2024-05-07T16:18:57.327Z] JavaTest Message: shutting down test
sxa commented 1 month ago

Which docker static is a good one to compare? We do have some tests failures are related with docker ones, which should not be the issue with the os, arch platform. So I think it's fair to limit the comparison to docker static ones.

test-docker-ubuntu2004-x64-2 was the one I had a clean run on fir my 22+35 tests so should be a good stable choice.

smlambert commented 1 month ago

jdk_lang

[2024-05-07T16:37:21.389Z] STDOUT:
[2024-05-07T16:37:21.389Z] Thread[#25,ForkJoinPool-1-worker-1,5,CarrierThreads]
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:621)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.System$2.parkVirtualThread(System.java:2652)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[2024-05-07T16:37:21.390Z]     java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.park(TracePinnedThreads.java:52)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.lambda$testPinnedCausedBySynchronizedBlock$0(TracePinnedThreads.java:68) <== monitors:1
[2024-05-07T16:37:21.390Z]     jdk.test.lib.thread.VThreadRunner.lambda$run$0(VThreadRunner.java:66)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
[2024-05-07T16:37:21.390Z] 
[2024-05-07T16:37:21.390Z] hello
[2024-05-07T16:37:21.390Z] hello
[2024-05-07T16:37:21.390Z] hello
[2024-05-07T16:37:21.390Z] hello
[2024-05-07T16:37:21.390Z] Thread[#29,ForkJoinPool-1-worker-2,5,CarrierThreads]
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:621)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.System$2.parkVirtualThread(System.java:2652)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[2024-05-07T16:37:21.390Z]     java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.park(TracePinnedThreads.java:52)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.invokePark(Native Method)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.lambda$testPinnedCausedByNativeMethod$1(TracePinnedThreads.java:81)
[2024-05-07T16:37:21.390Z]     jdk.test.lib.thread.VThreadRunner.lambda$run$0(VThreadRunner.java:66)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
[2024-05-07T16:37:21.390Z] 
[2024-05-07T16:37:21.390Z] Thread[#29,ForkJoinPool-1-worker-2,5,CarrierThreads]
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:621)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.System$2.parkVirtualThread(System.java:2652)
[2024-05-07T16:37:21.390Z]     java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[2024-05-07T16:37:21.390Z]     java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.park(TracePinnedThreads.java:52)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads$1C.<clinit>(TracePinnedThreads.java:93)
[2024-05-07T16:37:21.390Z]     TracePinnedThreads.lambda$testPinnedCausedByClassInitializer$2(TracePinnedThreads.java:96)
[2024-05-07T16:37:21.390Z]     jdk.test.lib.thread.VThreadRunner.lambda$run$0(VThreadRunner.java:66)
[2024-05-07T16:37:21.390Z]     java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
[2024-05-07T16:37:21.390Z] 
[2024-05-07T16:37:21.390Z] STDERR:
[2024-05-07T16:37:21.390Z] STARTED    TracePinnedThreads::testPinnedCausedBySynchronizedBlock 'testPinnedCausedBySynchronizedBlock()'
[2024-05-07T16:37:21.390Z] org.opentest4j.AssertionFailedError: reason:MONITOR not found!!! ==> expected: <true> but was: <false>
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
[2024-05-07T16:37:21.390Z]  at TracePinnedThreads.assertContains(TracePinnedThreads.java:168)
[2024-05-07T16:37:21.390Z]  at TracePinnedThreads.testPinnedCausedBySynchronizedBlock(TracePinnedThreads.java:71)
[2024-05-07T16:37:21.390Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-05-07T16:37:21.390Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-05-07T16:37:21.390Z]  at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
[2024-05-07T16:37:21.390Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
[2024-05-07T16:37:21.390Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.391Z]  at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.391Z]  at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
[2024-05-07T16:37:21.391Z]  at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:142)
[2024-05-07T16:37:21.391Z]  at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
[2024-05-07T16:37:21.391Z]  at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:61)
[2024-05-07T16:37:21.391Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-05-07T16:37:21.391Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-05-07T16:37:21.391Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2024-05-07T16:37:21.391Z]  at java.base/java.lang.Thread.run(Thread.java:1583)
[2024-05-07T16:37:21.391Z] FAILED     TracePinnedThreads::testPinnedCausedBySynchronizedBlock 'testPinnedCausedBySynchronizedBlock()'
[2024-05-07T16:37:21.391Z] STARTED    TracePinnedThreads::testContention 'testContention()'
[2024-05-07T16:37:21.391Z] SUCCESSFUL TracePinnedThreads::testContention 'testContention()'
[2024-05-07T16:37:21.391Z] STARTED    TracePinnedThreads::testPinnedCausedByNativeMethod 'testPinnedCausedByNativeMethod()'
[2024-05-07T16:37:21.391Z] org.opentest4j.AssertionFailedError: reason:NATIVE not found!!! ==> expected: <true> but was: <false>
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
[2024-05-07T16:37:21.391Z]  at TracePinnedThreads.assertContains(TracePinnedThreads.java:168)
[2024-05-07T16:37:21.391Z]  at TracePinnedThreads.testPinnedCausedByNativeMethod(TracePinnedThreads.java:82)
[2024-05-07T16:37:21.391Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-05-07T16:37:21.391Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-05-07T16:37:21.391Z]  at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
[2024-05-07T16:37:21.391Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
[2024-05-07T16:37:21.392Z]  at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.392Z]  at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.392Z]  at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
[2024-05-07T16:37:21.392Z]  at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
[2024-05-07T16:37:21.392Z]  at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:142)
[2024-05-07T16:37:21.392Z]  at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
[2024-05-07T16:37:21.392Z]  at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:61)
[2024-05-07T16:37:21.392Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-05-07T16:37:21.392Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-05-07T16:37:21.392Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2024-05-07T16:37:21.392Z]  at java.base/java.lang.Thread.run(Thread.java:1583)
[2024-05-07T16:37:21.392Z] FAILED     TracePinnedThreads::testPinnedCausedByNativeMethod 'testPinnedCausedByNativeMethod()'
sxa commented 1 month ago

@smlambert Are these failures already known about? I'm just nervous that I didn't see any of them in my sanity/extended runs on jdk-22+35 (So, of course, they could be specific to 21).

Also I see that the RHEL7 test was done on the latest build (build-scripts/jobs/jdk21u/jdk21u-linux-x64-temurin 183) where the docker ones seem to have all been done on the last release but with the master branch of aqa-tests so they may not be in sync with the material, as opposed to the pass on RHEL being related to it not running in a container.

What's your feeling on this @smlambert (since you're working through them based on the comments!)? I'm inclined to suggest either running again with v1.0.1-release branch, or against the latest nightly to clear up the results log a bit.

sxa commented 1 month ago

jdk_lang InfoTest - sleep not where its expected to be by the openjdk test calling it

Which job/machine was that from @smlambert? The Amazon Linux 2023 x64 machine seems to have it in /usr/bin so it should have been available without any issues.

sophia-guo commented 1 month ago

Tests unstable on docker containers https://github.com/adoptium/aqa-tests/issues/2138

sophia-guo commented 1 month ago

sanity.openjdk on https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D2/

https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/202/ - master aqa-tests with release jdk - failed https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/206/ - release branch with release jdk -passed.

Rerun failed java_lang and java_security with release branch and release jdk on amazon x64 jdk21 https://ci.adoptium.net/job/Grinder/10007/ - Failed as three failed tests are not in https://github.com/adoptium/jdk21u/releases/tag/jdk-21.0.3-ga rerun with failed 5 test cases https://ci.adoptium.net/job/Grinder/10008/ - Only one test java/lang/ProcessHandle/InfoTest.java failed, same as aarch64 - sleep issue

22:32:21  test InfoTest.test3(): failure
22:32:21  java.lang.AssertionError: Command: expected: 'sleep', actual: /usr/bin/coreutils expected [true] but found [false]

extended.openjdk on https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D2/ https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/56/console https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/59/) - with release branch

on https://ci.adoptium.net/computer/test%2Ddocker%2Damazon2023%2Dx64%2D1/ -with release branch https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/62/ - hotspot_serviceability_jvmti passed. cancelled to rerun the failure targets -jvm_compiler,jdk_other,jdk_net,jdk_security3,jdk_tools ( failing might due to the mismatch of the tests and jdk) https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/63/

smlambert commented 1 month ago

Which job/machine was that from @smlambert? The Amazon Linux 2023 x64 machine seems to have it in /usr/bin so it should have been available without any issues.

Given only one machine is configured, test-docker-amazon2023-x64-1, its that one.

Output from the test: Command: expected: 'sleep', actual: /usr/bin/coreutils expected [true] but found [false]

sxa commented 1 month ago

Which job/machine was that from @smlambert? The Amazon Linux 2023 x64 machine seems to have it in /usr/bin so it should have been available without any issues.

Given only one machine is configured, test-docker-amazon2023-x64-1, its that one.

To be clear, we've been running the tests on an amazon linux x64 and an aarch64 machine.

Output from the test: Command: expected: 'sleep', actual: /usr/bin/coreutils expected [true] but found [false]

OK yeah that's one we've seen before. I guess we don't have the upstream fix referred to in https://github.com/adoptium/infrastructure/issues/2886#issuecomment-2007502126 but the OS choosing to use a different implementation for the sleep command should definitely be considered non-blocking for the purposes of this.

sxa commented 1 month ago

Tests unstable on docker containers #2138

@sophia-guo It would be good to go through that list and see how many are still valid. While that lists quite a few failing tests it was last updated over two years ago and I don't think we're seeing many failures that are specific to docker containers now.

sxa commented 1 month ago

sanity.openjdk on https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D2/

https://ci.adoptium.net/job/Test_openjdk21_hs_sanity.openjdk_x86-64_linux/202/console -waiting

extended.openjdk on https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D2/ https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/56/console - waiting

Noting that similar to my earler comment these both seem to have been run with the master branch of aqa-tests and a release build, which means there is a good chance of higher numbers of failures due to the test material being out of sync with the product under test. Diagnosis and conclusions will be easier if we have them consistent.

sxa commented 1 month ago

I've kicked off an extended.openjdk run with v1.0.1-release on the x64 machine at https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_linux/57

Also kicked off some re-runs including jdk_lang and jdk_security from the AL2023 sanity run sanity run (with the sleep / coreutils issue) on aarch64 but I don't expect this to show anything too useful (I initially thought that Fedora39 would be the same, but I was mistaken and it uses a normal coreutils package for sleep so won't show the same issues, so I added a run on CentOS8):

On AL2023 aarch64 machine:

bash-5.2# cat /usr/bin/sleep
#!/usr/bin/coreutils --coreutils-prog-shebang=sleep
bash-5.2# rpm -qf /usr/bin/sleep
coreutils-single-8.32-30.amzn2023.0.3.aarch64
bash-5.2#

On CentOS8:

[root@ee6bf41532c6 /]# rpm -qf `which sleep`
coreutils-single-8.30-12.el8.aarch64
[root@ee6bf41532c6 /]# cat /usr/bin/sleep
#!/usr/bin/coreutils --coreutils-prog-shebang=sleep
[root@ee6bf41532c6 /]#
sxa commented 1 month ago

Keeping the aarch64 machine busy with re-runs with v1.0.1-release but I don't expect to have the time to triage them...

NOTE: The intermittent finalizer test on JDK21 extended.openjdk is a Unexpected Debuggee Exception: java.lang.OutOfMemoryError at line 145 - on the basis of a discussion yesterday I tried this on the TC machines and it is intermittent although at a slightly different frequency. See runs 4328 (aarch64 - 27/55 failed so far) and 4329 (x64 - 14/100 failed) but it has been variable in different high iteration runs.

sxa commented 1 month ago

jdk_lang

  • java/lang/String/CompactString/NegativeSize.java - OutOfMemoryError
[2024-05-07T16:18:57.327Z] JavaTest Message: Test threw exception: java.lang.RuntimeException: Failed: Expected OutOfMemoryError

Noting that this is not an OutOfMemoryError causing the test to fail (Therefore is unlikely to be related to insufficient resources on the machine) but is a failure due to the absence of such an error.

EDIT: I've run Grinders 9969 through 9974 on several machines, some docker, some non-docker, some with many iterations, and also grinders 4323-4325 on the Temurin Compliance machines with the same release build and v1.0.1-release material. 100% failure rate on this test so it's not an intermittent failure.

NOTE: For some reason the ones after 9974 don't seem to have a "Test Results" link but from the console log they all failed too

EDIT2: The ones after 9974, and the ones on the TC server were run with the v1.0.1-release branch and couldn't locate the test (although showed an unstable rather than failure result) so it is likely that this failure is the result of using incompatible test material and not a product or machine issue: [2024-05-14T07:52:22.033Z] Error: Cannot find file: /home/jenkins/workspace/Grinder/aqa-tests/TKG/../openjdk/openjdk-jdk/test/jdk/java/lang/String/CompactString/NegativeSize.java

sxa commented 1 month ago

jdk_lang

* java/lang/Thread/virtual/TracePinnedThreads.java - `org.opentest4j.AssertionFailedError: reason:NATIVE not found!!! ==> expected: <true> but was: <false>`

* Are some of these tests failing because they need the native test library and its not being provided?

Unclear as I don't have the job any more, but it did not fail in the Grinder from the v1.0.1-release branch run which was also using "releases" so I would expect that setup to have automatically pulled the correct things across. Seems most likely another case of incorrect test material, although it may be worth a multi-iteration x64 Grinder to see if it's intermittent so:

sxa commented 1 month ago

JDK8 extended.openjdk Single failure in sun/security/tools/keytool/autotest.sh.autotest - retry 100 iterations on docker-CentOS8 (All passed)

autotest.sh ``` [2024-05-14T01:56:04.806Z] The command result should be OK, but it's not. Try run the command manually and type these input into it: [2024-05-14T01:56:04.806Z] [2024-05-14T01:56:04.806Z] ERR is: [2024-05-14T01:56:04.806Z] [2024-05-14T01:56:04.806Z] OUT is: [2024-05-14T01:56:04.806Z] keytool error: java.io.IOException: load failed [2024-05-14T01:56:04.806Z] [2024-05-14T01:56:04.806Z] Exception in thread "main" java.io.IOException: load failed [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.P11KeyStore.engineLoad(P11KeyStore.java:764) [2024-05-14T01:56:04.806Z] at java.security.KeyStore.load(KeyStore.java:1445) [2024-05-14T01:56:04.806Z] at sun.security.tools.keytool.Main.doCommands(Main.java:869) [2024-05-14T01:56:04.806Z] at sun.security.tools.keytool.Main.run(Main.java:378) [2024-05-14T01:56:04.806Z] at sun.security.tools.keytool.Main.main(Main.java:371) [2024-05-14T01:56:04.806Z] at KeyToolTest.test(KeyToolTest.java:159) [2024-05-14T01:56:04.806Z] at KeyToolTest.test(KeyToolTest.java:127) [2024-05-14T01:56:04.806Z] at KeyToolTest.testOK(KeyToolTest.java:182) [2024-05-14T01:56:04.806Z] at KeyToolTest.testPKCS11(KeyToolTest.java:525) [2024-05-14T01:56:04.806Z] at KeyToolTest.main(KeyToolTest.java:1345) [2024-05-14T01:56:04.806Z] Caused by: javax.security.auth.login.LoginException [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.SunPKCS11.login(SunPKCS11.java:1333) [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.P11KeyStore.login(P11KeyStore.java:864) [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.P11KeyStore.engineLoad(P11KeyStore.java:751) [2024-05-14T01:56:04.806Z] ... 9 more [2024-05-14T01:56:04.806Z] Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_USER_TYPE_INVALID [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.wrapper.PKCS11.C_Login(Native Method) [2024-05-14T01:56:04.806Z] at sun.security.pkcs11.SunPKCS11.login(SunPKCS11.java:1317) [2024-05-14T01:56:04.806Z] ... 11 more ```

Testing on two other relatively new distributions: Fedora 39 (Passing ok) and Ubuntu 24.04 - (Passing ok - sophiaguo)

sophia- Testing autotest.sh only on test-docker-ubuntu2204-armv8l-2 https://ci.adoptium.net/job/Grinder/10006/ - [100 iteration] - (Failed)

JDK17 extended.openjdk Two encryption test failures: sun/security/pkcs11/ec/TestCurves.java.TestCurves and sun/security/pkcs11/ec/TestECDSA.java.TestECDSA - retry 100 iterations on docker-ubuntu2204-2 - (Passing ok - sophiaguo)

TestCurves ``` [2024-05-14T05:05:42.387Z] CLASSPATH=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestCurves.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/jtreg.jar \ [2024-05-14T05:05:42.387Z] /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java \ [2024-05-14T05:05:42.387Z] -Dtest.vm.opts='-ea -esa -Xmx512m -XX:+UseCompressedOops' \ [2024-05-14T05:05:42.387Z] -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-XX:+UseCompressedOops' \ [2024-05-14T05:05:42.387Z] -Dtest.compiler.opts= \ [2024-05-14T05:05:42.387Z] -Dtest.java.opts= \ [2024-05-14T05:05:42.387Z] -Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:42.387Z] -Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:42.387Z] -Dtest.timeout.factor=8.0 \ [2024-05-14T05:05:42.387Z] -Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \ [2024-05-14T05:05:42.387Z] -Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \ [2024-05-14T05:05:42.387Z] -Dtest.name=sun/security/pkcs11/ec/TestCurves.java \ [2024-05-14T05:05:42.387Z] -Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestCurves.java \ [2024-05-14T05:05:42.387Z] -Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec \ [2024-05-14T05:05:42.387Z] -Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ [2024-05-14T05:05:42.387Z] -Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestCurves.d \ [2024-05-14T05:05:42.387Z] -Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestCurves.d:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:42.387Z] -Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestCurves.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:42.388Z] -Dtest.modules=jdk.crypto.cryptoki/sun.security.pkcs11.wrapper \ [2024-05-14T05:05:42.388Z] --add-modules jdk.crypto.cryptoki \ [2024-05-14T05:05:42.388Z] --add-exports jdk.crypto.cryptoki/sun.security.pkcs11.wrapper=ALL-UNNAMED \ [2024-05-14T05:05:42.388Z] -ea \ [2024-05-14T05:05:42.388Z] -esa \ [2024-05-14T05:05:42.388Z] -Xmx512m \ [2024-05-14T05:05:42.388Z] -XX:+UseCompressedOops \ [2024-05-14T05:05:42.388Z] -Djava.library.path=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \ [2024-05-14T05:05:42.388Z] com.sun.javatest.regtest.agent.MainWrapper /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/sun/security/pkcs11/ec/TestCurves.d/main.0.jta [2024-05-14T05:05:42.388Z] [2024-05-14T05:05:42.388Z] ACTION: build -- Passed. All files up to date [2024-05-14T05:05:42.388Z] REASON: Named class compiled on demand [2024-05-14T05:05:42.388Z] TIME: 0.0 seconds [2024-05-14T05:05:42.388Z] messages: [2024-05-14T05:05:42.388Z] command: build TestCurves [2024-05-14T05:05:42.388Z] reason: Named class compiled on demand [2024-05-14T05:05:42.388Z] started: Tue May 14 05:05:39 UTC 2024 [2024-05-14T05:05:42.388Z] finished: Tue May 14 05:05:39 UTC 2024 [2024-05-14T05:05:42.388Z] elapsed time (seconds): 0.0 [2024-05-14T05:05:42.388Z] [2024-05-14T05:05:42.388Z] ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.Exception: Signature did not verify ```
TestECDSA ``` [2024-05-14T05:05:44.033Z] PATH=/bin:/usr/bin:/usr/sbin \ [2024-05-14T05:05:44.033Z] /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/javac \ [2024-05-14T05:05:44.033Z] -J-ea \ [2024-05-14T05:05:44.033Z] -J-esa \ [2024-05-14T05:05:44.033Z] -J-Xmx512m \ [2024-05-14T05:05:44.033Z] -J-XX:+UseCompressedOops \ [2024-05-14T05:05:44.033Z] -J-Dtest.vm.opts='-ea -esa -Xmx512m -XX:+UseCompressedOops' \ [2024-05-14T05:05:44.033Z] -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-XX:+UseCompressedOops' \ [2024-05-14T05:05:44.033Z] -J-Dtest.compiler.opts= \ [2024-05-14T05:05:44.033Z] -J-Dtest.java.opts= \ [2024-05-14T05:05:44.034Z] -J-Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:44.034Z] -J-Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \ [2024-05-14T05:05:44.034Z] -J-Dtest.timeout.factor=8.0 \ [2024-05-14T05:05:44.034Z] -J-Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \ [2024-05-14T05:05:44.034Z] -J-Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \ [2024-05-14T05:05:44.034Z] -J-Dtest.name=sun/security/pkcs11/ec/TestECDSA.java \ [2024-05-14T05:05:44.034Z] -J-Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestECDSA.java \ [2024-05-14T05:05:44.034Z] -J-Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec \ [2024-05-14T05:05:44.034Z] -J-Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d \ [2024-05-14T05:05:44.034Z] -J-Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -J-Dtest.modules=jdk.crypto.cryptoki \ [2024-05-14T05:05:44.034Z] --add-modules jdk.crypto.cryptoki \ [2024-05-14T05:05:44.034Z] -d /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d \ [2024-05-14T05:05:44.034Z] -sourcepath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ [2024-05-14T05:05:44.034Z] -classpath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11/ec/TestECDSA.d:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/java/security/testlibrary:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/test/lib:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17156558239020/jdk_security3_0/work/classes/2/sun/security/pkcs11 /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestECDSA.java [2024-05-14T05:05:44.034Z] direct: [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:780: warning: [removal] Policy in java.security has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] Policy.setPolicy(null); // Clear the policy created by JIB if any [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:799: warning: [removal] SecurityManager in java.lang has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(new SecurityManager()); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:799: warning: [removal] setSecurityManager(SecurityManager) in System has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(new SecurityManager()); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/PKCS11Test.java:811: warning: [removal] setSecurityManager(SecurityManager) in System has been deprecated and marked for removal [2024-05-14T05:05:44.034Z] System.setSecurityManager(null); [2024-05-14T05:05:44.034Z] ^ [2024-05-14T05:05:44.034Z] Note: /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib/jdk/test/lib/artifacts/ArtifactResolver.java uses or overrides a deprecated API. [2024-05-14T05:05:44.034Z] Note: Recompile with -Xlint:deprecation for details. [2024-05-14T05:05:44.034Z] Note: /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/lib/jdk/test/lib/artifacts/JibArtifactManager.java uses unchecked or unsafe operations. [2024-05-14T05:05:44.034Z] Note: Recompile with -Xlint:unchecked for details. [2024-05-14T05:05:44.034Z] 4 warnings [2024-05-14T05:05:44.034Z] [2024-05-14T05:05:44.034Z] ACTION: main -- Failed. Execution failed: `main' threw exception: java.lang.Exception: Sign/verify 3 failed [2024-05-14T05:05:44.034Z] REASON: User specified action: run main/othervm TestECDSA ```
sophia-guo commented 4 weeks ago

Also triggered x64 jdk17, 8 - waiting for machine , need to confirm if it's necessary? @jiekang

09:49:27  PROC: Nsanity readline: [GSSLibStub:97] libgssapi.so: cannot open shared object file: No such file or directory
09:49:27  PROC: Nsanity readline: [GSSLibStub:66] [GSSLibStub_init] libName=libgssapi_krb5.so
09:49:27  PROC: Nsanity readline: [GSSLibStub:97] libgssapi_krb5.so: cannot open shared object file: No such file or directory
0
sophia-guo commented 4 weeks ago

Summary for amazon x64 jdk21 - ✔️

jdk8, jdk17, jdk11 waiting for result https://github.com/adoptium/aqa-tests/issues/5300#issuecomment-2113409321

Summary for aarch64: https://github.com/adoptium/aqa-tests/issues/5300#issuecomment-2108492474

@jiekang

Haroon-Khel commented 4 weeks ago

ref https://github.com/adoptium/aqa-tests/issues/5300#issuecomment-2113409321

We're seeing sun/security/krb5/auto/rcache_usemd5.sh fail on other platforms, jdk8 arm32 linux https://github.com/adoptium/infrastructure/issues/3043#issuecomment-2115082200

jiekang commented 4 weeks ago

sun/security/krb5/auto/rcache_usemd5.sh has a related test fix that was backported to JDK 8 last month. It will be in the July CPU. You may be able to ignore it for now.

https://github.com/openjdk/jdk8u-dev/commit/6b53212ef78ad50f9eede829c5ff87cadcdb434b https://bugs.openjdk.org/browse/JDK-8258855

@Haroon-Khel @sophia-guo

sophia-guo commented 3 weeks ago

Latest status jdk21:

jdk 17:

jdk11:

jdk 8:

sun/security/pkcs11/ec/TestCurves is common issue for jdk17, 11 and 8 with aarch64 and linux64, will open an issue to investigate

sun/security/tools/keytool/autotest.sh is common issue for jdk8 linux64 and aarch64.

All failed tests are jdk_security tests. @jiekang

Haroon-Khel commented 3 weeks ago

Repeating the same failures using Corretto jdk

jdk17

linux x64 sun/security/pkcs11/ec/TestCurves.java https://ci.adoptium.net/job/Grinder/10053/console - 10 out 10 failed aarch64 sun/security/pkcs11/ec/TestCurves.java.TestCurves - 1 out 10 passed sun/security/pkcs11/ec/TestECDSA.java.TestECDSA - 1 out 10 passed https://ci.adoptium.net/job/Grinder/10054/console

jdk11

linux x64 jdk_security3 https://ci.adoptium.net/job/Grinder/10055/console

jdk8

linux x64 jdk_security3 https://ci.adoptium.net/job/Grinder/10056/console aarch64 sun/security/tools/keytool/autotest.sh https://ci.adoptium.net/job/Grinder/10057/console

All using Sophia's v1.0.1-release branch

sophia-guo commented 3 weeks ago

https://github.com/adoptium/aqa-tests/issues/5300#issuecomment-2124681919

Confirmed Corretto has same failures with jdk_security3.

jiekang commented 3 weeks ago

Thank you @sophia-guo and @Haroon-Khel !

Haroon-Khel commented 3 weeks ago

In https://ci.adoptium.net/job/Grinder/10053/testReport/, jdk17 corretto linux x64, we have 1 pass out of 10 iterations

sun/security/pkcs11/ec/TestECDSA.java.TestECDSA

Beginning test run TestECDSA...
===> testNSS: Starting test run
nssLibDir: /usr/lib64/
libsoftokn3 version not found, set to 0.0: /usr/lib64/libsoftokn3.so
Running test with provider SunPKCS11-NSS (security manager disabled) ...
     secp192r1: Unsupported: Key Length: EC key must be at least 256 bits. The specific key size 192 is not supported
     sect163r1: Unsupported: Key Length: EC key must be at least 256 bits. The specific key size 163 is not supported
     sect571r1: Unsupported: Key Length: EC key must be at most 521 bits. The specific key size 571 is not supported
Passed
Passed
Passed
Passed
Testing with default signature format: ASN.1

But still has a standard error of java.lang.Exception: Sign/verify 3 failed

    at TestECDSA.testSigning(TestECDSA.java:250)
    at TestECDSA.test(TestECDSA.java:197)
    at TestECDSA.main(TestECDSA.java:168)
    at PKCS11Test.premain(PKCS11Test.java:805)
    at PKCS11Test.testNSS(PKCS11Test.java:481)
    at PKCS11Test.main(PKCS11Test.java:181)
    at TestECDSA.main(TestECDSA.java:119)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
    at java.base/java.lang.Thread.run(Thread.java:840)

JavaTest Message: Test threw exception: java.lang.Exception: Sign/verify 3 failed
JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.Exception: Sign/verify 3 failed

Not sure why the test is passed

sun/security/pkcs11/ec/TestCurves.java.TestCurves

Beginning test run TestCurves...
===> testNSS: Starting test run
nssLibDir: /usr/lib64/
libsoftokn3 version not found, set to 0.0: /usr/lib64/libsoftokn3.so
Running test with provider SunPKCS11-NSS (security manager disabled) ...
Finding supported curves using list from SunEC

     secp256r1: Supported
     secp384r1: Supported
     secp521r1: Supported
Testing secp256r1 [NIST P-256,X9.62 prime256v1] (1.2.840.10045.3.1.7)...
  SHA1withECDSA  SHA224withECDSA  SHA256withECDSA  SHA384withECDSA  SHA512withECDSA
Testing secp384r1 [NIST P-384] (1.3.132.0.34)...
  SHA1withECDSA  SHA224withECDSA  SHA256withECDSA  SHA384withECDSA  SHA512withECDSA
Testing secp521r1 [NIST P-521] (1.3.132.0.35)...
  SHA1withECDSA  SHA224withECDSA  SHA256withECDSA  SHA384withECDSA  SHA512withECDSA
OK
Completed test with provider SunPKCS11-NSS (322 ms).
testNSS: Completed
jiekang commented 3 weeks ago

Without prior expertise, this NSS fuzziness could probably be a multi-week investigation of it's own :|

smlambert commented 3 weeks ago

Those security testcases mention the recommendation for using the latest NSS version, given https://wiki.mozilla.org/NSS:Release_Versions that would be 3.100 and that jives with seeing those tests pass on OS's with v3.98. If Amazon Linux 2023 can only be updated to 3.90, we may consider those failing tests as not necessarily valid for this platform.

As long as this is documented and understood (for Zeke when in discussion with others), I think we can/should mark this as acceptable. Ideally the upstream test code would include a check that said, if NSS version is less than X, then skip or mark as invalid against lesser versions.

sophia-guo commented 3 weeks ago

Triage done, close it.

sxa commented 1 week ago

Some extra notes:

It's possible that RHEL has some backports of fixes from later libnss3 versions, but it does mean there is at least some doubt as to whether 3.90 in AL2023 is the cause of the failures.

sxa commented 1 week ago

Ref libnss3:

While this would suggest we're ok for this particular test, we know that the master branch is not suitable for testing the GA level as per the original attempts to run from the aqa-tests master branch. Here are some runs with the nightly build and the master branch (same set as per this comment):

running infotest JDK17 on multiple machines as grinder 10235 through 10239 - all passed extended.openjdk other than java/lang/ProcessBuilder/PipelineLeaksFD.java.PipelineLeaksFD so the sleep problem does not occur there. It is a known problem with CentOS and UBI too.

And x64:

[*] - The multiple runs which show a 15% pass rate on the AL2023/aarch64 machine used the command in the twisty below, and one passing and one failing run output is shown below:

Command line used ``` CLASSPATH=/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/test/lib:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11:/home/sxa/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/sxa/jvmtest/openjdk/jtreg/lib/jtreg.jar \ /home/sxa/jdk-17.0.11+9/bin/java \ -Dtest.vm.opts='-ea -esa -Xmx512m' \ -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m' \ -Dtest.compiler.opts= \ -Dtest.java.opts= \ -Dtest.jdk=/home/sxa/jdk-17.0.11+9 \ -Dcompile.jdk=/home/sxa/jdk-17.0.11+9 \ -Dtest.timeout.factor=8.0 \ -Dtest.root=/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk \ -Dtest.name=sun/security/pkcs11/ec/TestCurves.java \ -Dtest.file=/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec/TestCurves.java \ -Dtest.src=/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec \ -Dtest.src.path=/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11 \ -Dtest.classes=/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d \ -Dtest.class.path=/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/test/lib:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11 \ -Dtest.class.path.prefix=/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d:/home/sxa/aqa-tests/openjdk/openjdk-jdk/test/jdk/sun/security/pkcs11/ec:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/test/lib:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11 \ -Dtest.modules=jdk.crypto.cryptoki/sun.security.pkcs11.wrapper \ --add-modules jdk.crypto.cryptoki \ --add-exports jdk.crypto.cryptoki/sun.security.pkcs11.wrapper=ALL-UNNAMED \ -ea \ -esa \ -Xmx512m \ -Djava.security.manager=allow \ com.sun.javatest.regtest.agent.MainWrapper /home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/sun/security/pkcs11/ec/TestCurves.d/main.1.jta sm ```
Passing output ``` Beginning test run TestCurves... ===> testNSS: Starting test run nssLibDir: /usr/lib64/ libsoftokn3 version not found, set to 0.0: /usr/lib64/libsoftokn3.so WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by PKCS11Test (file:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d/) WARNING: Please consider reporting this to the maintainers of PKCS11Test WARNING: System::setSecurityManager will be removed in a future release Running test with provider SunPKCS11-NSS (security manager enabled) ... Finding supported curves using list from SunEC secp256r1: Supported secp384r1: Supported secp521r1: Supported Testing secp256r1 [NIST P-256,X9.62 prime256v1] (1.2.840.10045.3.1.7)... SHA1withECDSA SHA224withECDSA SHA256withECDSA SHA384withECDSA SHA512withECDSA Testing secp384r1 [NIST P-384] (1.3.132.0.34)... SHA1withECDSA SHA224withECDSA SHA256withECDSA SHA384withECDSA SHA512withECDSA Testing secp521r1 [NIST P-521] (1.3.132.0.35)... SHA1withECDSA SHA224withECDSA SHA256withECDSA SHA384withECDSA SHA512withECDSA OK Completed test with provider SunPKCS11-NSS (473 ms). testNSS: Completed STATUS:Passed. ```
Failing output ``` Beginning test run TestCurves... ===> testNSS: Starting test run nssLibDir: /usr/lib64/ libsoftokn3 version not found, set to 0.0: /usr/lib64/libsoftokn3.so WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by PKCS11Test (file:/home/sxa/aqa-tests/TKG/output_17175900395804/jdk_custom_0/work/classes/0/sun/security/pkcs11/ec/TestCurves.d/) WARNING: Please consider reporting this to the maintainers of PKCS11Test WARNING: System::setSecurityManager will be removed in a future release Running test with provider SunPKCS11-NSS (security manager enabled) ... Finding supported curves using list from SunEC secp256r1: Supported secp384r1: Supported secp521r1: Supported Testing secp256r1 [NIST P-256,X9.62 prime256v1] (1.2.840.10045.3.1.7)... SHA1withECDSA SHA224withECDSA SHA256withECDSA SHA384withECDSA SHA512withECDSA Testing secp384r1 [NIST P-384] (1.3.132.0.34)... SHA1withECDSA SHA224withECDSA SHA256withECDSA SHA384withECDSA SHA512withECDSA Testing secp521r1 [NIST P-521] (1.3.132.0.35)... SHA1withECDSA SHA224withECDSAjava.lang.Exception: Signature did not verify at TestCurves.testSigning(TestCurves.java:121) at TestCurves.main(TestCurves.java:84) at PKCS11Test.premain(PKCS11Test.java:805) at PKCS11Test.testNSS(PKCS11Test.java:481) at PKCS11Test.main(PKCS11Test.java:181) at TestCurves.main(TestCurves.java:50) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138) at java.base/java.lang.Thread.run(Thread.java:840) JavaTest Message: Test threw exception: java.lang.Exception: Signature did not verify JavaTest Message: shutting down test STATUS:Failed.`main' threw exception: java.lang.Exception: Signature did not verify ```

By comparison the sun/security/pkcs11/ec/TestCurves.java and sun/security/pkcs11/ec/TestECDSA.java tests do seem to pass consistently elsewhere:

Based on earlier comments about failures in JDK11/x64 I'm also trying the following runs:

sxa commented 6 days ago

Summary for aarch64

Most of the failures seen on the branch in the PKCS11 tests do not occur on the latest nightly and latest test code, indicating that changes in the test code may have resolved some of the issues after the last release which we have been testing on. There are still failures in autotest.sh in JDK8's extended.openjdk although they also occur on Fedora 39 (Not on 35) so that is likely more pervasive and not specific to AL2023. The sleep failure in the InfoTest test is due to the use of coreutils (like UBI and CentOS) which makes the sleep command a small wrapper which starts #!/usr/bin/coreutils so the process listing shows coreutils instead of sleep (Ref: https://github.com/adoptium/infrastructure/issues/2886)

Summary for x64

testJcmdDump had failed but that test seems to show intermittent failures and is not believed to be specific to Amazon. The InfoTest/sleep failure is as mentioned in the aarch64 summary above. JDK8 e/o is also showing a number of pkcs11 and nio (character set related) failures which are not showing on aarch64. Re-running 50 iterations of all the failures on three static docker containers on the same host: AL2023, Ubuntu2204-7, Debian12 (pkcs failures only on AL2023, nio on all three) ... running 5000 iterations over the weekend on AL2023 at https://ci.adoptium.net/job/Grinder/10271/) ... running 5000 iterations over the weekend on AL2023 at https://ci.adoptium.net/job/Grinder/10271/)

Why the differences?

The pkcs11 failures on JDK8/x64 do not occur on aarch64 because there is code that explicitly skips it on Linux/aarch64 - it reports Unsupported OS, skipping: Linux-aarch64-64 due to this map not having an entry for Linux-aarch64-64 so it triggers this line.

Related issues: