search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

jdk19 OpenJDK java/lang/Thread/virtual/stress/Skynet.java hang / timeout #16728

Closed pshipton closed 9 months ago

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/111/ jdk_lang_0 java/lang/Thread/virtual/stress/Skynet.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/111/openjdk_test_output.tar.gz

21:55:35  ACTION: main -- Error. Program `/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 2400000ms, elapsed time including timeout handling was 2404113ms).
21:55:35  REASON: User specified action: run main/othervm/timeout=300 --enable-preview Skynet 
21:55:35  TIME:   2404.116 seconds
21:55:35  messages:
21:55:35  command: main --enable-preview Skynet
21:55:35  reason: User specified action: run main/othervm/timeout=300 --enable-preview Skynet 
21:55:35  Mode: othervm [/othervm specified]
21:55:35  Timeout information:
21:55:35  Running jcmd on process 2529589
21:55:35  Dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/aqa-tests/TKG/output_1676426825596/jdk_lang_0/work/scratch/2/core.20230214.215530.2529589.0001.dmp
21:55:35  Dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/aqa-tests/TKG/output_1676426825596/jdk_lang_0/work/scratch/2/javacore.20230214.215531.2529589.0002.txt
21:55:35  Running jstack on process 2529589
21:55:35  2023-02-14T21:55:32.813679073
21:55:35  Virtual machine: 2529589 JVM information:
21:55:35  JRE 19 Linux aarch64-64-Bit Compressed References 20230215_180 (JIT enabled, AOT enabled)
21:55:35  OpenJ9   - 711ca6c2adb
21:55:35  OMR      - 6adea0343aa
21:55:35  JCL      - 47925e78c4e based on jdk-19.0.2+7
21:55:35  
21:55:35  "main" prio=5 Id=2 WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.join(Thread.java:2151)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.join(Thread.java:2227)
21:55:35    at app//com.sun.javatest.regtest.agent.MainWrapper.main(MainWrapper.java:74)
21:55:35  
21:55:35  "JIT Compilation Thread-000" prio=10 Id=4 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-001 Suspended" prio=10 Id=5 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-002 Suspended" prio=10 Id=6 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-003 Suspended" prio=10 Id=7 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-004 Suspended" prio=10 Id=8 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-005 Suspended" prio=10 Id=9 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-006 Suspended" prio=10 Id=10 RUNNABLE
21:55:35  
21:55:35  "JIT Diagnostic Compilation Thread-007 Suspended" prio=10 Id=11 RUNNABLE
21:55:35  
21:55:35  "JIT-SamplerThread" prio=10 Id=12 TIMED_WAITING
21:55:35  
21:55:35  "IProfiler" prio=5 Id=13 RUNNABLE
21:55:35  
21:55:35  "Common-Cleaner" prio=8 Id=3 TIMED_WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:137)
21:55:35    at java.base@19.0.2-internal/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.run(Thread.java:1573)
21:55:35    at java.base@19.0.2-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
21:55:35  
21:55:35  "Concurrent Mark Helper" prio=1 Id=14 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=15 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=16 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=17 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=18 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=19 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=20 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=21 RUNNABLE
21:55:35  
21:55:35  "Attach API wait loop" prio=10 Id=24 RUNNABLE
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.IPC.waitSemaphore(Native Method)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.CommonDirectory.waitSemaphore(CommonDirectory.java:264)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.WaitLoop.waitForNotification(WaitLoop.java:66)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.WaitLoop.run(WaitLoop.java:157)
21:55:35  
21:55:35  "Finalizer thread" prio=5 Id=25 RUNNABLE
21:55:35  
21:55:35  "MainThread" prio=5 Id=27 WAITING
21:55:35    at java.base@19.0.2-internal/jdk.internal.misc.Unsafe.park(Native Method)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
21:55:35    at app//Skynet$Channel.receive(Skynet.java:110)
21:55:35    at app//Skynet.skynet(Skynet.java:60)
21:55:35    at app//Skynet.main(Skynet.java:50)
21:55:35    at java.base@19.0.2-internal/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
21:55:35    at java.base@19.0.2-internal/java.lang.reflect.Method.invoke(Method.java:578)
21:55:35    at app//com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.run(Thread.java:1573)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-2" prio=5 Id=31 RUNNABLE
21:55:35    at java.base@19.0.2-internal/jdk.internal.vm.Continuation.enterImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/jdk.internal.vm.Continuation.run(Continuation.java:198)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:218)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-3" prio=5 Id=41 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiUnmountEnd(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.afterYield(VirtualThread.java:396)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:223)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-6" prio=5 Id=47 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiUnmountEnd(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.afterTerminate(VirtualThread.java:424)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:221)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-7" prio=5 Id=48 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiMountBegin(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:215)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "file lock watchdog" prio=10 Id=10700077 TIMED_WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.util.TimerThread.mainLoop(Timer.java:569)
21:55:35    at java.base@19.0.2-internal/java.util.TimerThread.run(Timer.java:522)
21:55:35  
21:55:35  "Attachment portNumber: 40951" prio=10 Id=10708739 RUNNABLE
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.dumpAllThreadsImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.getThreadInfo(DiagnosticUtils.java:245)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.executeDiagnosticCommand(DiagnosticUtils.java:181)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.Attachment.doCommand(Attachment.java:248)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.Attachment.run(Attachment.java:159)
21:55:35  
21:55:35  
21:55:35  --- Timeout information end.
21:55:35  elapsed time (seconds): 2404.116
21:55:35  configuration:
21:55:35  STDOUT:
21:55:35  Result: 499999500000 in 396129 ms
21:55:35  Result: 499999500000 in 141690 ms
21:55:35  Result: 499999500000 in 344331 ms
21:55:35  Result: 499999500000 in 326083 ms
21:55:35  Result: 499999500000 in 300962 ms
21:55:35  Result: 499999500000 in 379486 ms
21:55:35  Result: 499999500000 in 222160 ms
21:55:35  Result: 499999500000 in 194157 ms
21:55:35  Result: 499999500000 in 50598 ms
21:55:35  Timeout refired 2400 times
amicic commented 1 year ago

Based on one run that I have logs for, RI does not show non-determinism in object liveness. We can see at the end of global GCs (red markings), what are valid points for estimating live set since all dead objects are collected (barring effects of floating garbage due to concurrent GC), there is consistently only about 50MB of live objects.

image

For easier comparisons, I'm repeating the same J9 graph from before but visualized by an external same tool (GCEasy) same one used for RI. Clearly, a lot variations at global GC points (Another valid question is why we have so many of them, and it's both due to larger observed live set, but also due to our relatively passive heap expansion).

image
pshipton commented 1 year ago

@babsingh @fengxue-IS Pls create a release note for the known issues with Loom and any workarounds. https://github.com/eclipse-openj9/openj9/blob/master/doc/release-notes/0.37/0.37.md

pshipton commented 1 year ago

More Invalid JIT return address running TimedGet and Skynet testing. https://github.com/eclipse-openj9/openj9/issues/17163#issuecomment-1506275869

pshipton commented 1 year ago

We're also seeing this assert 18:54:01 22:52:19.743 0x100238df600 j9vm.224 * ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19u/jdk19u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/vm/ContinuationHelpers.cpp:276: ((!VM_VMHelpers::isConcurrentlyScanned(continuationState) && ((0) == VM_VMHelpers::getCarrierThread(continuationState))))

pshipton commented 1 year ago

These grinders are all running just the Skynet test, rather than all jdk_lang testing.

Skynet grinder on xlinux, -Xnocompressedrefs - passed 100 runs

plinux grinder 100x timeout in https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_0/2422

AIX grinder 50x Invalid JIT return address https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_3/1841/

pshipton commented 1 year ago

We'll continue to investigate these issues for jdk20.

amicic commented 1 year ago

The recent assert, even though the same test, should probably be a separate issue.

The timeout in pLinux has a core dump - could we get it?

babsingh commented 1 year ago

The timeout in pLinux has a core dump - could we get it?

plinux timeout: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_0/2422

Core from plinux: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Grinder_iteration_0/2422/openjdk_test_output.tar.gz

pshipton commented 1 year ago

The timeout in pLinux has a core dump - could we get it?

pshipton commented 1 year ago

Created https://github.com/eclipse-openj9/openj9/issues/17181 for the assert.

babsingh commented 1 year ago

@fengxue-IS Can we close this issue based on our recent perf changes? No failures have been seen for the past two months.

tajila commented 1 year ago

We can de-prioritze this since we have imporved performance, but I'd like to keep this open to track the fact the J9 requires a larger heap

vij-singh commented 1 year ago

What's the plan for this one? Are we planning anything for 0.42?

babsingh commented 1 year ago

What's the plan for this one? Are we planning anything for 0.42?

This is not a blocker since we can increase the heap size by changing the -Xmx value.

For 0.42 (Java 21), we are targeting the work to relocate unmounted continuations above 4G:

Note: The above two issues can potentially be merged into one.

@fengxue-IS is working on the above feature.

In the future, there might be more GC features to further optimize heap utilization. So, this issue will most likely stay open and be pushed forward.

JasonFengJ9 commented 1 year ago

Adding two latest JDK21 failures.

JDK21 ppc64_aix(p8-java1-ibm11)

openjdk version "21-internal" 2023-09-19
OpenJDK Runtime Environment (build 21-internal-adhoc.****.BuildJDK21ppc64aixNightly)
Eclipse OpenJ9 VM (build master-c46b26096a8, JRE 21 AIX ppc64-64-Bit Compressed References 20230901_53 (JIT enabled, AOT enabled)
OpenJ9   - c46b26096a8
OMR      - 75d233597f4
JCL      - fce06129139 based on jdk-21+35)

[2023-09-01T23:05:15.050Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode150
[2023-09-01T23:05:15.050Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops 

[2023-09-02T00:36:28.672Z] TEST: java/lang/Thread/virtual/stress/Skynet.java#default

[2023-09-02T00:36:28.673Z] STDERR:
[2023-09-02T00:36:28.673Z] Exception in thread "" Exception in thread "" Exception in thread "" java.lang.StackOverflowError
[2023-09-02T00:36:28.673Z]  at java.base/java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:839)
[2023-09-02T00:36:28.673Z]  at Skynet$Channel.send(Skynet.java:103)
[2023-09-02T00:36:28.673Z]  at Skynet.skynet(Skynet.java:77)
[2023-09-02T00:36:28.673Z]  at Skynet.lambda$skynet$1(Skynet.java:82)
[2023-09-02T00:36:28.673Z]  at java.base/java.lang.VirtualThread.run(VirtualThread.java:317)
[2023-09-02T00:36:28.673Z]  at java.base/jdk.internal.vm.Continuation.enter(Continuation.java:184)
[2023-09-02T00:36:28.676Z] TEST RESULT: Error. Program `/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 2400000ms, elapsed time including timeout handling was 3143517ms).
[2023-09-02T00:36:28.676Z] --------------------------------------------------
[2023-09-02T00:45:16.178Z] Test results: passed: 918; error: 1
[2023-09-02T00:45:26.128Z] Report written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_16936089829867/jdk_lang_0/report/html/report.html
[2023-09-02T00:45:26.128Z] Results written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_16936089829867/jdk_lang_0/work
[2023-09-02T00:45:26.128Z] Error: Some tests failed or other problems occurred.
[2023-09-02T00:45:26.128Z] -----------------------------------
[2023-09-02T00:45:26.128Z] jdk_lang_0_FAILED

[2023-09-02T00:45:26.128Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode501
[2023-09-02T00:45:26.128Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs 

[2023-09-02T01:31:19.363Z] TEST: java/lang/Thread/virtual/stress/Skynet.java#default

[2023-09-02T01:31:19.364Z] STDERR:
[2023-09-02T01:31:19.364Z] Unhandled exception
[2023-09-02T01:31:19.364Z] Type=Segmentation error vmState=0x00000000
[2023-09-02T01:31:19.364Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033
[2023-09-02T01:31:19.364Z] Handler1=09001000A22BA400 Handler2=09001000A20F0430
[2023-09-02T01:31:19.364Z] R0=0000000000001808 R1=0000010024130EB0 R2=09001000A22BE1F8 R3=0000000000000000
[2023-09-02T01:31:19.364Z] R4=0000000000000218 R5=000000007F800000 R6=000000003ADBAD21 R7=09001000A1EDD368
[2023-09-02T01:31:19.364Z] R8=0000000000000000 R9=09001000A1EDD338 R10=0000000000000000 R11=0000000000000000
[2023-09-02T01:31:19.364Z] R12=FFFFFFFF84000244 R13=000001002413C800 R14=0000010031C395E0 R15=0000010023E12100
[2023-09-02T01:31:19.364Z] R16=0000010022806EF8 R17=000000000000000A R18=0A0000003BC275B8 R19=0A0000000078E868
[2023-09-02T01:31:19.364Z] R20=00000000000EBD4E R21=0000000000000000 R22=090000001D9BC37C R23=0A000200094E391C
[2023-09-02T01:31:19.364Z] R24=0000000000000000 R25=0000000000000001 R26=09001000A22B4EA8 R27=0000000000000000
[2023-09-02T01:31:19.364Z] R28=0000000000000103 R29=000001002E09E9A8 R30=0000000000000000 R31=0AA69804B09D413E
[2023-09-02T01:31:19.364Z] IAR=090000001CC98388 LR=090000001CC99B58 MSR=A00000000200D032 CTR=090000001CC98378
[2023-09-02T01:31:19.364Z] CR=2400024420000004 FPSCR=8202000000000000 XER=2000000482020000
[2023-09-02T01:31:19.364Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR2 bf5b75a420000000 (f: 536870912.000000, d: -1.675997e-03)
[2023-09-02T01:31:19.364Z] FPR3 3f5b75a420000000 (f: 536870912.000000, d: 1.675997e-03)
[2023-09-02T01:31:19.364Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-09-02T01:31:19.364Z] FPR5 40a7060000000000 (f: 0.000000, d: 2.947000e+03)
[2023-09-02T01:31:19.364Z] FPR6 4070b1f20d4afc0c (f: 223018000.000000, d: 2.671216e+02)
[2023-09-02T01:31:19.364Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-09-02T01:31:19.364Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-09-02T01:31:19.364Z] FPR9 402791272ee9e400 (f: 787080192.000000, d: 1.178350e+01)
[2023-09-02T01:31:19.364Z] FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
[2023-09-02T01:31:19.364Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-09-02T01:31:19.364Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-09-02T01:31:19.364Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.364Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-02T01:31:19.365Z] Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
[2023-09-02T01:31:19.365Z] Module_base_address=090000001CA00000
[2023-09-02T01:31:19.365Z] Target=2_90_20230901_53 (AIX 7.2)
[2023-09-02T01:31:19.365Z] CPU=ppc64 (16 logical CPUs) (0x780000000 RAM)
[2023-09-02T01:31:19.365Z] ----------- Stack Backtrace -----------
[2023-09-02T01:31:19.365Z] (0x090000001CA8D4E8 [libj9vm29.so+0x8d4e8])
[2023-09-02T01:31:19.365Z] runJavaThread+0x28c (0x090000001CA74A50 [libj9vm29.so+0x74a50])
[2023-09-02T01:31:19.365Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000001CA02FF4 [libj9vm29.so+0x2ff4])
[2023-09-02T01:31:19.365Z] omrsig_protect+0x4fc (0x090000001CDF81E0 [libj9prt29.so+0x5f1e0])
[2023-09-02T01:31:19.365Z] javaThreadProc+0x70 (0x090000001CA02E94 [libj9vm29.so+0x2e94])
[2023-09-02T01:31:19.365Z] thread_wrapper+0x14c (0x090000001CE61590 [libj9thr29.so+0x5590])
[2023-09-02T01:31:19.365Z] _pthread_body+0xe8 (0x090000000056104C [libpthreads.a+0x404c])
[2023-09-02T01:31:19.365Z] ---------------------------------------
[2023-09-02T01:31:19.365Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/02 01:18:07 - please wait.

[2023-09-02T01:31:19.366Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
[2023-09-02T01:31:19.366Z] --------------------------------------------------
[2023-09-02T01:41:11.866Z] Test results: passed: 918; failed: 1
[2023-09-02T01:41:11.867Z] Report written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_16936089829867/jdk_lang_j9_0/report/html/report.html
[2023-09-02T01:41:11.867Z] Results written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_16936089829867/jdk_lang_j9_0/work
[2023-09-02T01:41:11.867Z] Error: Some tests failed or other problems occurred.
[2023-09-02T01:41:11.867Z] -----------------------------------
[2023-09-02T01:41:11.867Z] jdk_lang_j9_0_FAILED
pshipton commented 1 year ago

Also https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_1/45/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/45/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_Nightly_testList_0/46/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/45/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0/44/

Maybe a regression. @babsingh https://github.com/eclipse-openj9/openj9/compare/022a2a444...c46b26096a8 https://github.com/ibmruntimes/openj9-openjdk-jdk21/compare/d36e4341ce8f...fce06129139

pshipton commented 1 year ago

Since it seems to be failing in every build, we need to fix it or exclude it asap.

pshipton commented 1 year ago

In https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/46 we also have these failures. java/lang/Thread/virtual/stress/TimedGet.java.TimedGet java/lang/Thread/virtual/stress/YieldALot.java#id0.YieldALot_id0

babsingh commented 1 year ago

Since it seems to be failing in every build, we need to fix it or exclude it asap.

I will exclude it until further investigation is performed.

babsingh commented 1 year ago

fyi @fengxue-IS

babsingh commented 1 year ago

In https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/46 we also have these failures. java/lang/Thread/virtual/stress/TimedGet.java.TimedGet java/lang/Thread/virtual/stress/YieldALot.java#id0.YieldALot_id0

These failures seem like infra issues:

20:51:49  Cannot contact ub16p8j92: java.lang.InterruptedException
20:55:20  --------------------------------------------------
20:55:20  TEST: java/lang/Thread/virtual/stress/TimedGet.java
20:55:20  TEST JDK: /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/openjdkbinary/j2sdk-image
20:55:20  
20:55:20  ACTION: build -- Error. Cannot get VM for test: java.net.SocketTimeoutException: Accept timed out
...
20:55:20  TEST RESULT: Error. Cannot get VM for test: java.net.SocketTimeoutException: Accept timed out
20:55:20  --------------------------------------------------
21:25:20  TEST: java/lang/Thread/virtual/stress/YieldALot.java#id0
21:25:20  TEST JDK: /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/openjdkbinary/j2sdk-image
21:25:20  
21:25:20  ACTION: build -- Error. Cannot get VM for test: java.net.SocketTimeoutException: Accept timed out
babsingh commented 1 year ago

re https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1707149910: https://github.com/adoptium/aqa-tests/pull/4744 excludes Skynet.java#default.

tajila commented 1 year ago

The most suspect change is likely https://github.com/eclipse-openj9/openj9/commit/92cb9c95bbe6141a5e25e65355a85bf1ae24c569'

tajila commented 1 year ago

Has someone decoded the native stack?

JasonFengJ9 commented 1 year ago

JDK21 x86-64_linux(ubu22x86-svl-rt4-1) has better native stack

08:40:09  ===============================================
08:40:09  Running test serviceability_jvmti_j9_0 ...
08:40:09  ===============================================
08:40:09  serviceability_jvmti_j9_0 Start Time: Sun Sep  3 05:40:07 2023 Epoch Time (ms): 1693744807444
08:40:09  variation: Mode150
08:40:09  JVM_OPTIONS:  -XX:+UseCompressedOops 

08:45:39  TEST: serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#default

08:45:39  STDERR:
08:45:39  Unhandled exception
08:45:39  Type=Segmentation error vmState=0x00040000
08:45:39  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
08:45:39  Handler1=00007F6902B2B750 Handler2=00007F6902A81710 InaccessibleAddress=0000000000000010
08:45:39  RDI=0000000000000000 RSI=000000000000000C RAX=0000000000000000 RBX=00007F68E0208020
08:45:39  RCX=00007F6890060210 RDX=00007F68E0207B70 R8=0000000000000038 R9=00007F68FC02D5F0
08:45:39  R10=00007F6900ED8840 R11=0000000000000001 R12=0000000000000000 R13=0000000000000000
08:45:39  R14=00007F68E0206F80 R15=00007F68FC02D5F0
08:45:39  RIP=00007F6900ED960A GS=0000 FS=0000 RSP=00007F68E0206F50
08:45:39  EFlags=0000000000010246 CS=0033 RBP=0000000000245500 ERR=0000000000000004
08:45:39  TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000010
08:45:39  xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:39  xmm1 00000000ffef9908 (f: 4293892352.000000, d: 2.121465e-314)
08:45:39  xmm2 0000000000253469 (f: 2438249.000000, d: 1.204655e-317)
08:45:39  xmm3 00000000000b56d8 (f: 743128.000000, d: 3.671540e-318)
08:45:39  xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:39  xmm5 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:45:39  xmm6 bfa252f438e10c1e (f: 954272768.000000, d: -3.578914e-02)
08:45:39  xmm7 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:45:39  xmm8 000000000000000a (f: 10.000000, d: 4.940656e-323)
08:45:39  xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:39  xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:39  xmm11 0000015200000151 (f: 337.000000, d: 7.172346e-312)
08:45:39  xmm12 0000013d00000140 (f: 320.000000, d: 6.726727e-312)
08:45:39  xmm13 000001380000013f (f: 319.000000, d: 6.620627e-312)
08:45:39  xmm14 0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
08:45:39  xmm15 000001420000013b (f: 315.000000, d: 6.832826e-312)
08:45:39  Module=/home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
08:45:39  Module_base_address=00007F6900EC2000
08:45:39  Target=2_90_20230903_99 (Linux 5.15.0-78-generic)
08:45:39  CPU=amd64 (4 logical CPUs) (0x1f017d000 RAM)
08:45:39  ----------- Stack Backtrace -----------
08:45:39  followReferencesCallback+0xdca (0x00007F6900ED960A [libj9jvmti29.so+0x1760a])
08:45:39  _ZN23MM_ReferenceChainWalker6doSlotEPP8J9ObjectllS1_.localalias+0x96 (0x00007F69010B5706 [libj9gc29.so+0x38706])
08:45:39  walkStackFrames+0x11d0 (0x00007F6902B6F600 [libj9vm29.so+0x87600])
08:45:39  walkContinuationStackFrames.part.0+0x56 (0x00007F6902B87306 [libj9vm29.so+0x9f306])
08:45:39  _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP16J9VMContinuationPvPFvP8J9JavaVMPP8J9ObjectS4_P16J9StackWalkStatePKvEbb+0x3e (0x00007F69010C225E [libj9gc29.so+0x4525e])
08:45:39  _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x13a (0x00007F69010B956A [libj9gc29.so+0x3c56a])
08:45:39  _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xcf (0x00007F69010B7E2F [libj9gc29.so+0x3ae2f])
08:45:39  _ZN14MM_RootScanner12scanAllSlotsEP18MM_EnvironmentBase+0x3a (0x00007F69010BB5EA [libj9gc29.so+0x3e5ea])
08:45:39  j9gc_ext_reachable_objects_do+0x1c1 (0x00007F69010B74D1 [libj9gc29.so+0x3a4d1])
08:45:39  jvmtiFollowReferences+0x2ab (0x00007F6900EDA16B [libj9jvmti29.so+0x1816b])
08:45:39  Java_VThreadStackRefTest_test+0x163 (0x00007F68E0940723 [libVThreadStackRefTest.so+0x3723])
08:45:39  ffi_call_unix64+0x52 (0x00007F6902CF55DA [libj9vm29.so+0x20d5da])
08:45:39  ffi_call_int+0x1a1 (0x00007F6902CF4771 [libj9vm29.so+0x20c771])
08:45:39  _ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x11420 (0x00007F6902B98C20 [libj9vm29.so+0xb0c20])
08:45:39  bytecodeLoopCompressed+0xca (0x00007F6902B877EA [libj9vm29.so+0x9f7ea])
08:45:39   (0x00007F6902C82702 [libj9vm29.so+0x19a702])
08:45:39  ---------------------------------------
08:45:39  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 05:45:38 - please wait.

08:45:41  TEST: serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#no-vmcontinuations
08:45:41  STDERR:
08:45:41  Unhandled exception
08:45:41  Type=Segmentation error vmState=0x00040000
08:45:41  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
08:45:41  Handler1=00007F15A4A8A750 Handler2=00007F15A49E0710 InaccessibleAddress=0000000000000010
08:45:41  RDI=0000000000000000 RSI=000000000000000C RAX=0000000000000000 RBX=00007F157E163020
08:45:41  RCX=00007F152805E1D0 RDX=00007F157E162B70 R8=0000000000000038 R9=00007F15A002DDA0
08:45:41  R10=00007F159EDF4840 R11=0000000000000001 R12=0000000000000000 R13=0000000000000000
08:45:41  R14=00007F157E161F80 R15=00007F15A002DDA0
08:45:41  RIP=00007F159EDF560A GS=0000 FS=0000 RSP=00007F157E161F50
08:45:41  EFlags=0000000000010246 CS=0033 RBP=0000000000244D00 ERR=0000000000000004
08:45:41  TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000010
08:45:41  xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:41  xmm1 00000000ffdc06a8 (f: 4292609792.000000, d: 2.120831e-314)
08:45:41  xmm2 0000000000253469 (f: 2438249.000000, d: 1.204655e-317)
08:45:41  xmm3 00000000000b56d8 (f: 743128.000000, d: 3.671540e-318)
08:45:41  xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:41  xmm5 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:45:41  xmm6 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
08:45:41  xmm7 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:45:41  xmm8 000000000000000a (f: 10.000000, d: 4.940656e-323)
08:45:41  xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:41  xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:45:41  xmm11 0000015200000151 (f: 337.000000, d: 7.172346e-312)
08:45:41  xmm12 0000013d00000140 (f: 320.000000, d: 6.726727e-312)
08:45:41  xmm13 000001380000013f (f: 319.000000, d: 6.620627e-312)
08:45:41  xmm14 0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
08:45:41  xmm15 000001420000013b (f: 315.000000, d: 6.832826e-312)
08:45:41  Module=/home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
08:45:41  Module_base_address=00007F159EDDE000
08:45:41  Target=2_90_20230903_99 (Linux 5.15.0-78-generic)
08:45:41  CPU=amd64 (4 logical CPUs) (0x1f017d000 RAM)
08:45:41  ----------- Stack Backtrace -----------
08:45:41  followReferencesCallback+0xdca (0x00007F159EDF560A [libj9jvmti29.so+0x1760a])
08:45:41  _ZN23MM_ReferenceChainWalker6doSlotEPP8J9ObjectllS1_.localalias+0x96 (0x00007F159EFD1706 [libj9gc29.so+0x38706])
08:45:41  walkStackFrames+0x11d0 (0x00007F15A4ACE600 [libj9vm29.so+0x87600])
08:45:41  walkContinuationStackFrames.part.0+0x56 (0x00007F15A4AE6306 [libj9vm29.so+0x9f306])
08:45:41  _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP16J9VMContinuationPvPFvP8J9JavaVMPP8J9ObjectS4_P16J9StackWalkStatePKvEbb+0x3e (0x00007F159EFDE25E [libj9gc29.so+0x4525e])
08:45:41  _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x13a (0x00007F159EFD556A [libj9gc29.so+0x3c56a])
08:45:41  _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xcf (0x00007F159EFD3E2F [libj9gc29.so+0x3ae2f])
08:45:41  _ZN14MM_RootScanner12scanAllSlotsEP18MM_EnvironmentBase+0x3a (0x00007F159EFD75EA [libj9gc29.so+0x3e5ea])
08:45:41  j9gc_ext_reachable_objects_do+0x1c1 (0x00007F159EFD34D1 [libj9gc29.so+0x3a4d1])
08:45:41  jvmtiFollowReferences+0x2ab (0x00007F159EDF616B [libj9jvmti29.so+0x1816b])
08:45:41  Java_VThreadStackRefTest_test+0x163 (0x00007F15A4009723 [libVThreadStackRefTest.so+0x3723])
08:45:41  ffi_call_unix64+0x52 (0x00007F15A4C545DA [libj9vm29.so+0x20d5da])
08:45:41  ffi_call_int+0x1a1 (0x00007F15A4C53771 [libj9vm29.so+0x20c771])
08:45:41  _ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x11420 (0x00007F15A4AF7C20 [libj9vm29.so+0xb0c20])
08:45:41  bytecodeLoopCompressed+0xca (0x00007F15A4AE67EA [libj9vm29.so+0x9f7ea])
08:45:41   (0x00007F15A4BE1702 [libj9vm29.so+0x19a702])
08:45:41  ---------------------------------------
08:45:41  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 05:45:40 - please wait.

08:45:41  TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
08:45:41  --------------------------------------------------
08:48:08  Test results: passed: 151; failed: 3
08:48:08  Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/jvmtest/openjdk/report/html/report.html
08:48:08  Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/aqa-tests/TKG/output_16937447659990/serviceability_jvmti_j9_0/work
08:48:08  Error: Some tests failed or other problems occurred.
08:48:08  -----------------------------------
08:48:08  serviceability_jvmti_j9_0_FAILED
babsingh commented 1 year ago

The native stacks in https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1707409489 are related to https://github.com/eclipse-openj9/openj9/issues/17712. VThreadStackRefTest should be excluded; https://github.com/adoptium/aqa-tests/pull/4744 should correctly exclude it.

babsingh commented 1 year ago

Has someone decoded the native stack?

I looked at the diagnostic files for the Linux x64 failure: https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_x86-64_linux_Nightly_testList_0/46/openjdk_test_output.tar.gz.

No system dump files are generated in the build.

Details from the javacores

OutOfMemoryError: native memory exhausted is set in createContinuation.

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "ForkJoinPool-1-worker-1" J9VMThread:0x0000000000254300, omrthread_t:0x00007FF404000F98, java/lang/Thread:0x00000000C00EC970, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x18, isDaemon:true)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000C0040000)
3XMTHREADINFO1            (native thread ID:0x7F1, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x00007FF44A9F1000, to:0x00007FF44AA71000, size:0x80000)
3XMCPUTIME               CPU usage total: 595.021440137 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=1861728 (0x1C6860)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/internal/vm/Continuation.enterImpl(Native Method)
4XESTACKTRACE                at jdk/internal/vm/Continuation.run(Continuation.java:211(Compiled Code))
4XESTACKTRACE                at java/lang/VirtualThread.runContinuation(VirtualThread.java:229(Compiled Code))
4XESTACKTRACE                at java/lang/VirtualThread$$Lambda/0x000000004aa71a00.run(Bytecode PC:4(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doExec(ForkJoinTask.java:387(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.scan(ForkJoinPool.java:1843(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.runWorker(ForkJoinPool.java:1808)
4XESTACKTRACE                at java/util/concurrent/ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
3XMTHREADINFO3           Native callstack:
...
4XENATIVESTACK               triggerDumpAgents+0x615 (0x00007FF485991A65 [libj9dmp29.so+0x1fa65])
4XENATIVESTACK               rasDumpHookExceptionSysthrow+0x88 (0x00007FF48598FD38 [libj9dmp29.so+0x1dd38])
4XENATIVESTACK               J9HookDispatch+0x136 (0x00007FF491738926 [libj9hookable29.so+0x1926])
4XENATIVESTACK               internalSetCurrentExceptionWithCause+0x5ba (0x00007FF4902743CA [libj9vm29.so+0x383ca])
4XENATIVESTACK               setCurrentExceptionUTF+0x116 (0x00007FF490274886 [libj9vm29.so+0x38886])
4XENATIVESTACK               createContinuation+0x1e7 (0x00007FF4902D9BC7 [libj9vm29.so+0x9dbc7])
4XENATIVESTACK               enterContinuation+0x54b (0x00007FF4902DA52B [libj9vm29.so+0x9e52b])
4XENATIVESTACK               _ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x46aa (0x00007FF4902DFEAA [libj9vm29.so+0xa3eaa])
4XENATIVESTACK               bytecodeLoopCompressed+0xca (0x00007FF4902DB7EA [libj9vm29.so+0x9f7ea])
4XENATIVESTACK                (0x00007FF4903D6702 [libj9vm29.so+0x19a702])

gdb output from my local run

...
#7  0x00007f34a4b8a6bc in triggerDumpAgents (vm=0x7f34a0013f20, self=0x29b200, eventFlags=262144, eventData=0x7f347d851070) at /root/openj9-openjdk-jdk21/openj9/runtime/rasdump/trigger.c:1041
#8  0x00007f34a4b887d4 in rasDumpHookExceptionSysthrow (hookInterface=hookInterface@entry=0x7f34a0016240, eventNum=eventNum@entry=10, eventData=eventData@entry=0x7f347d8511b0, userData=userData@entry=0x0)
    at /root/openj9-openjdk-jdk21/openj9/runtime/rasdump/trigger.c:1551
#9  0x00007f34a54389d1 in J9HookDispatch (hookInterface=0x7f34a0016240, taggedEventNum=<optimized out>, eventData=0x7f347d8511b0) at /root/openj9-openjdk-jdk21/omr/util/hookable/hookable.cpp:235
#10 0x00007f34a54774f8 in internalSetCurrentExceptionWithCause (currentThread=0x29b200, exceptionNumber=<optimized out>, detailMessage=<optimized out>, utfMessage=<optimized out>, cause=<optimized out>)
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/exceptionsupport.c:746
#11 0x00007f34a547788b in setCurrentExceptionWithUtfCause (currentThread=<optimized out>, exceptionNumber=<optimized out>, detailMessage=<optimized out>, utfMessage=<optimized out>, cause=<optimized out>)
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/exceptionsupport.c:807
#12 0x00007f34a5477a0e in setCurrentExceptionUTF (vmThread=0x29b200, exceptionNumber=33554443, detailUTF=0x7f34a56a3105 "native memory exhausted")
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/exceptionsupport.c:65
#13 0x00007f34a54e0e70 in createContinuation (currentThread=currentThread@entry=0x29b200, continuationObject=continuationObject@entry=0xcaf3d0b0)
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/ContinuationHelpers.cpp:86
#14 0x00007f34a54e183b in enterContinuation (currentThread=0x29b200, continuationObject=0xcaf3d0b0) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/ContinuationHelpers.cpp:182
#15 0x00007f34a54ede6e in VM_BytecodeInterpreterCompressed::enterContinuationImpl (_pc=<optimized out>, _sp=<optimized out>, this=<optimized out>)
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/BytecodeInterpreter.hpp:5396
#16 VM_BytecodeInterpreterCompressed::run (this=0x11a68, this@entry=0x7f347d8518d0, vmThread=0x0) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/BytecodeInterpreter.hpp:10911
#17 0x00007f34a54e2b37 in bytecodeLoopCompressed (currentThread=<optimized out>) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/BytecodeInterpreter.inc:112
#18 0x00007f34a561fb72 in c_cInterpreter () at /root/openj9-openjdk-jdk21/build/linux-x86_64-server-release/vm/runtime/vm/xcinterp.s:158
#19 0x00007f34a545f82e in runJavaThread (currentThread=0x1, currentThread@entry=0x29b200) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/callin.cpp:682
#20 0x00007f34a54de325 in javaProtectedThreadProc (portLibrary=portLibrary@entry=0x7f34a572f420 <j9portLibrary>, entryarg=entryarg@entry=0x29b200) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/vmthread.cpp:2104
#21 0x00007f34a5279370 in omrsig_protect (portLibrary=0x7f34a572f420 <j9portLibrary>, fn=0x7f34a54de290 <javaProtectedThreadProc(J9PortLibrary*, void*)>, fn_arg=0x29b200,
    handler=0x7f34a5482fa0 <structuredSignalHandler>, handler_arg=0x29b200, flags=506, result=0x7f347d851da8) at /root/openj9-openjdk-jdk21/omr/port/unix/omrsignal.c:425
#22 0x00007f34a54da7f2 in javaThreadProc (entryarg=0x7f34a0013f20) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/vmthread.cpp:383
#23 0x00007f34a5428dc7 in thread_wrapper (arg=0x7f3408010910) at /root/openj9-openjdk-jdk21/omr/thread/common/omrthread.c:1724
#24 0x00007f34a57d2609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#25 0x00007f34a592e133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) l ContinuationHelpers.cpp:86
81  #else
82  #define VMTHR_INITIAL_STACK_SIZE vm->stackSize
83  #endif
84
85          if ((stack = allocateJavaStack(vm, VMTHR_INITIAL_STACK_SIZE, NULL)) == NULL) {
86              vm->internalVMFunctions->setNativeOutOfMemoryError(currentThread, 0, 0);
87              j9mem_free_memory(continuation);
88              result = FALSE;
89              goto end;
90          }
babsingh commented 1 year ago

The most suspect change is likely https://github.com/eclipse-openj9/openj9/commit/92cb9c95bbe6141a5e25e65355a85bf1ae24c569'

https://github.com/eclipse-openj9/openj9/pull/18060 shouldn't cause this issue. Skynet.java#L27 relies upon vm.continuations (test flag). Prior to last week, vm.continuations was set to false by default. So, the Skynet test wasn't being run in our builds for the past year. Last Friday, https://github.com/ibmruntimes/openj9-openjdk-jdk21/pull/35 set the vm.continuations flag to true by default. This enabled the Skynet test in our builds in addition to all the other disabled virtual thread tests that depend on vm.continuations. It is hard to say when and what broke the test.

babsingh commented 1 year ago

Skynet Results

With OpenJ9 -Xcompressedrefs, the first iteration took ~9 mins whereas all iterations are less than 6 seconds with the RI and OpenJ9 -Xnocompressedrefs. The issue is related to allocating memory below 4G. Before, we had resolved this issue through the stack caching solution. @fengxue-IS Can you please verify if our stack caching solution still works as expected?

fengxue-IS commented 1 year ago

The perf we are getting from -Xcompressedrefs is expected due to JDK20's update on scheduler fairness, though my testing from before did not expose the OOM issue

0xdaryl commented 12 months ago

Hi... Given the current direction of the investigation, is there still a reason to tag this with comp:jit ?

babsingh commented 12 months ago

is there still a reason to tag this with comp:jit ?

No, I have removed comp:jit.

babsingh commented 11 months ago

@tajila This issue can be removed from the Java 21 milestone once the below PRs are merged:

babsingh commented 11 months ago

All four PRs in https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1751104856 have been merged. Skynet has been enabled. Moving this issue to the 0.43 release. fyi @TobiAjila

pshipton commented 11 months ago

Since Skynet is enabled, I assume the test excluded label can be removed.

pshipton commented 11 months ago

Skynet crash

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_1/70 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_1/70/openjdk_test_output.tar.gz

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/70 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/70/openjdk_test_output.tar.gz

Skynet failures

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/76 https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_1/76

babsingh commented 11 months ago

There is a mix of VM and GC issues; the failure is only seen on AIX and pLinux. @fengxue-IS @LinHu2016 Can you perform the initial diagnose? We might need to disable the test on AIX and pLinux if the fix requires time.

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_1/70/consoleFull

19:59:10  Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
19:59:10  Module_base_address=09000000075A4000
19:59:10  Target=2_90_20231017_87 (AIX 7.1)
19:59:10  CPU=ppc64 (16 logical CPUs) (0x780000000 RAM)
19:59:10  ----------- Stack Backtrace -----------
19:59:10  (0x09000000076317E8 [libj9vm29.so+0x8d7e8])
19:59:10  runJavaThread+0x28c (0x0900000007618D50 [libj9vm29.so+0x74d50])
19:59:10  _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x09000000075A6FF4 [libj9vm29.so+0x2ff4])
19:59:10  omrsig_protect+0x4fc (0x09000000070541E0 [libj9prt29.so+0x5f1e0])
19:59:10  javaThreadProc+0x70 (0x09000000075A6E94 [libj9vm29.so+0x2e94])
19:59:10  thread_wrapper+0x14c (0x09000000070BD590 [libj9thr29.so+0x5590])
19:59:10  _pthread_body+0xf0 (0x090000000051FE14 [libpthreads.a+0x3e14])
19:59:10  ---------------------------------------

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/70/consoleFull

19:57:15  Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
19:57:15  Module_base_address=090000001CED7000
19:57:15  Target=2_90_20231017_87 (AIX 7.1)
19:57:15  CPU=ppc64 (16 logical CPUs) (0x780000000 RAM)
19:57:15  ----------- Stack Backtrace -----------
19:57:15  _Z17stackSlotIteratorP8J9JavaVMPP8J9ObjectPvP16J9StackWalkStatePKv+0x34 (0x090000001CFA2FD8 [libj9gc_full29.so+0xcbfd8])
19:57:15  _ZL28vmThreadStackDoOSlotIteratorP10J9VMThreadP16J9StackWalkStatePP8J9ObjectPKv+0x38 (0x090000001CFA801C [libj9gc_full29.so+0xd101c])
19:57:15  IPRA.$walkMethodFrame+0x4a4 (0x090000001AAD7428 [libj9vm29.so+0x10b428])
19:57:15  walkStackFrames+0x2e0 (0x090000001AAD6464 [libj9vm29.so+0x10a464])
19:57:15  _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0xc4 (0x090000001CFA8208 [libj9gc_full29.so+0xd1208])
19:57:15  _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x22c (0x090000001CFA3610 [libj9gc_full29.so+0xcc610])
19:57:15  _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xe0 (0x090000001CFA3144 [libj9gc_full29.so+0xcc144])
19:57:15  _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0xd0 (0x090000001CFA6514 [libj9gc_full29.so+0xcf514])
19:57:15  _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x33c (0x090000001D03FC40 [libj9gc_full29.so+0x168c40])
19:57:15  _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x10 (0x090000001D06CD14 [libj9gc_full29.so+0x195d14])
19:57:15  _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x230 (0x090000001D0C9C34 [libj9gc_full29.so+0x1f2c34])
19:57:15  _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x1d8 (0x090000001D0C98DC [libj9gc_full29.so+0x1f28dc])
19:57:15  omrsig_protect+0x4fc (0x090000001ADBE1E0 [libj9prt29.so+0x5f1e0])
19:57:15  dispatcher_thread_proc+0x50 (0x090000001D0C99B4 [libj9gc_full29.so+0x1f29b4])
19:57:15  thread_wrapper+0x14c (0x090000001AE27590 [libj9thr29.so+0x5590])
19:57:15  _pthread_body+0xf0 (0x090000000051FE14 [libpthreads.a+0x3e14])

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/76/consoleFull https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_1/76/consoleFull

// Not much details are available
19:39:21  TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
pshipton commented 11 months ago

We might need to disable the test on AIX and pLinux if the fix requires time.

We should just go ahead and disable, otherwise the tests will be failing every night and weekend.

babsingh commented 11 months ago

We should just go ahead and disable, otherwise the tests will be failing every night and weekend.

Opened https://github.com/adoptium/aqa-tests/pull/4822 to disable the test.

fengxue-IS commented 11 months ago

I wasn't able to find a corefile in the test archive, re-running test in grinder to analyze AIX issue.

fengxue-IS commented 11 months ago

Locally re-run the test adding -Xint option to isolate issue on AIX machine, and I got a crash due to NativeOutOfMemoryError. will add more debug symbol with personal build to dig further.

pshipton commented 10 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0/89/ - win2019-x64-3 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0/89/openjdk_test_output.tar.gz

20:57:49  ACTION: main -- Error. Agent error: java.lang.Exception: Agent 84 timed out with a timeout of 960 seconds; check console log for any additional details
20:57:49  REASON: User specified action: run main PinALot 500000 
20:57:49  TIME:   1025.568 seconds
20:57:49  messages:
20:57:49  command: main PinALot 500000
20:57:49  reason: User specified action: run main PinALot 500000 
20:57:49  started: Mon Nov 06 20:40:37 EST 2023
20:57:49  Mode: agentvm
20:57:49  Agent id: 84
20:57:49  Timeout signalled after 960 seconds
20:57:49  Timeout information:
20:57:49  Running jcmd on process 7732
20:57:49  Dump written to F:\Users\jenkins\workspace\Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16993169074126\jdk_lang_0\work\scratch\2\core.20231106.205656.7732.0001.dmp
20:57:49  Dump written to F:\Users\jenkins\workspace\Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16993169074126\jdk_lang_0\work\scratch\2\javacore.20231106.205724.7732.0002.txt
20:57:49  Running jstack on process 7732
20:57:49  2023-11-06T20:57:25.490773600
20:57:49  Virtual machine: 7732 JVM information:
20:57:49  JRE 21 Windows Server 2019 amd64-64-Bit Compressed References 20231106_100 (JIT enabled, AOT enabled)
20:57:49  OpenJ9   - 132d734355a
20:57:49  OMR      - 2a9a7f4e479
20:57:49  JCL      - 07bee59fc7e based on jdk-21.0.1+12
20:57:49  
20:57:49  "main" prio=5 Id=2 WAITING
20:57:49    at java.base@21.0.1-internal/java.lang.Object.waitImpl(Native Method)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:255)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:221)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.join(Thread.java:2105)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.join(Thread.java:2181)
20:57:49    at app//com.sun.javatest.regtest.agent.MainActionHelper.runClass(MainActionHelper.java:219)
20:57:49    at app//com.sun.javatest.regtest.agent.AgentServer.doMain(AgentServer.java:317)
20:57:49    at app//com.sun.javatest.regtest.agent.AgentServer.run(AgentServer.java:242)
20:57:49    at app//com.sun.javatest.regtest.agent.AgentServer.main(AgentServer.java:71)
20:57:49  
20:57:49  "JIT Compilation Thread-000" prio=10 Id=4 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-001 Suspended" prio=10 Id=5 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-002 Suspended" prio=10 Id=6 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-003 Suspended" prio=10 Id=7 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-004 Suspended" prio=10 Id=8 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-005 Suspended" prio=10 Id=9 RUNNABLE
20:57:49  
20:57:49  "JIT Compilation Thread-006 Suspended" prio=10 Id=10 RUNNABLE
20:57:49  
20:57:49  "JIT Diagnostic Compilation Thread-007 Suspended" prio=10 Id=11 RUNNABLE
20:57:49  
20:57:49  "JIT-SamplerThread" prio=10 Id=12 TIMED_WAITING
20:57:49  
20:57:49  "IProfiler" prio=5 Id=13 RUNNABLE
20:57:49  
20:57:49  "Common-Cleaner" prio=8 Id=3 TIMED_WAITING
20:57:49    at java.base@21.0.1-internal/java.lang.Object.waitImpl(Native Method)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:255)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:221)
20:57:49    at java.base@21.0.1-internal/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:137)
20:57:49    at java.base@21.0.1-internal/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:57:49    at java.base@21.0.1-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
20:57:49  
20:57:49  "Finalizer thread" prio=5 Id=14 RUNNABLE
20:57:49  
20:57:49  "Concurrent Mark Helper" prio=1 Id=15 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=16 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=17 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=18 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=19 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=20 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=21 RUNNABLE
20:57:49  
20:57:49  "GC Worker" prio=5 Id=22 RUNNABLE
20:57:49  
20:57:49  "Attach API wait loop" prio=10 Id=25 RUNNABLE
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.IPC.waitSemaphore(Native Method)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.CommonDirectory.waitSemaphore(CommonDirectory.java:264)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.WaitLoop.waitForNotification(WaitLoop.java:66)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.WaitLoop.run(WaitLoop.java:157)
20:57:49  
20:57:49  "pool-1-thread-1" prio=5 Id=26 TIMED_WAITING
20:57:49    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:57:49  
20:57:49  "AgentVMThread" prio=5 Id=27 TIMED_WAITING
20:57:49    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:756)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1126)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:276)
20:57:49    at java.base@21.0.1-internal/java.lang.VirtualThread.joinNanos(VirtualThread.java:832)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.join(Thread.java:2219)
20:57:49    at PinALot.main(PinALot.java:65)
20:57:49    at java.base@21.0.1-internal/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
20:57:49    at java.base@21.0.1-internal/java.lang.reflect.Method.invoke(Method.java:580)
20:57:49    at app//com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:57:49  
20:57:49  "ForkJoinPool-1-worker-1" prio=5 Id=29 WAITING
20:57:49    at java.base@21.0.1-internal/jdk.internal.vm.Continuation.run(Continuation.java:213)
20:57:49    at java.base@21.0.1-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:227)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
20:57:49  
20:57:49  "VirtualThread-unparker" prio=5 Id=30 RUNNABLE
20:57:49    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
20:57:49    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
20:57:49    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:57:49    at java.base@21.0.1-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
20:57:49  
20:57:49  "file lock watchdog" prio=10 Id=32 TIMED_WAITING
20:57:49    at java.base@21.0.1-internal/java.lang.Object.waitImpl(Native Method)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:255)
20:57:49    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:221)
20:57:49    at java.base@21.0.1-internal/java.util.TimerThread.mainLoop(Timer.java:570)
20:57:49    at java.base@21.0.1-internal/java.util.TimerThread.run(Timer.java:523)
20:57:49  
20:57:49  "Attachment portNumber: 52666" prio=10 Id=34 RUNNABLE
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.dumpAllThreadsImpl(Native Method)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.getThreadInfo(DiagnosticUtils.java:249)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.executeDiagnosticCommand(DiagnosticUtils.java:185)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.Attachment.doCommand(Attachment.java:248)
20:57:49    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.Attachment.run(Attachment.java:159)
20:57:49  
20:57:49  
20:57:49  --- Timeout information end.
pshipton commented 10 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0/90/ - win2019-x64-3

20:23:58  ACTION: main -- Error. Agent error: java.lang.Exception: Agent 89 timed out with a timeout of 960 seconds; check console log for any additional details
20:23:58  REASON: User specified action: run main PinALot 500000 
20:23:58  TIME:   1026.441 seconds
20:23:58  messages:
20:23:58  command: main PinALot 500000
20:23:58  reason: User specified action: run main PinALot 500000 
20:23:58  started: Tue Nov 07 20:06:48 EST 2023
20:23:58  Mode: agentvm
20:23:58  Agent id: 89
20:23:58  Timeout signalled after 960 seconds
20:23:58  Timeout information:
20:23:58  Running jcmd on process 4472
20:23:58  Dump written to F:\Users\jenkins\workspace\Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16994012552318\jdk_lang_0\work\scratch\2\core.20231107.202308.4472.0001.dmp
20:23:58  Dump written to F:\Users\jenkins\workspace\Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16994012552318\jdk_lang_0\work\scratch\2\javacore.20231107.202337.4472.0002.txt
20:23:58  Running jstack on process 4472
20:23:58  2023-11-07T20:23:38.156186800
20:23:58  Virtual machine: 4472 JVM information:
20:23:58  JRE 21 Windows Server 2019 amd64-64-Bit Compressed References 20231107_101 (JIT enabled, AOT enabled)
20:23:58  OpenJ9   - 70f56555a97
20:23:58  OMR      - 2a9a7f4e479
20:23:58  JCL      - f8d29432229 based on jdk-21.0.1+12
20:23:58  
20:23:58  "main" prio=5 Id=2 RUNNABLE
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.Net.poll(Native Method)
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191)
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:280)
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
20:23:58    at java.base@21.0.1-internal/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
20:23:58    at java.base@21.0.1-internal/java.net.Socket$SocketInputStream.read(Socket.java:1099)
20:23:58    at java.base@21.0.1-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)
20:23:58    at java.base@21.0.1-internal/java.io.BufferedInputStream.implRead(BufferedInputStream.java:325)
20:23:58    at java.base@21.0.1-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:312)
20:23:58    at java.base@21.0.1-internal/java.io.FilterInputStream.read(FilterInputStream.java:71)
20:23:58    at app//com.sun.javatest.regtest.agent.AgentServer.run(AgentServer.java:236)
20:23:58    at app//com.sun.javatest.regtest.agent.AgentServer.main(AgentServer.java:71)
20:23:58  
20:23:58  "JIT Compilation Thread-000" prio=10 Id=4 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-001 Suspended" prio=10 Id=5 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-002 Suspended" prio=10 Id=6 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-003 Suspended" prio=10 Id=7 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-004 Suspended" prio=10 Id=8 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-005 Suspended" prio=10 Id=9 RUNNABLE
20:23:58  
20:23:58  "JIT Compilation Thread-006 Suspended" prio=10 Id=10 RUNNABLE
20:23:58  
20:23:58  "JIT Diagnostic Compilation Thread-007 Suspended" prio=10 Id=11 RUNNABLE
20:23:58  
20:23:58  "JIT-SamplerThread" prio=10 Id=12 TIMED_WAITING
20:23:58  
20:23:58  "IProfiler" prio=5 Id=13 RUNNABLE
20:23:58  
20:23:58  "Common-Cleaner" prio=8 Id=3 TIMED_WAITING
20:23:58    at java.base@21.0.1-internal/java.lang.Object.waitImpl(Native Method)
20:23:58    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:255)
20:23:58    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:221)
20:23:58    at java.base@21.0.1-internal/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:137)
20:23:58    at java.base@21.0.1-internal/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
20:23:58    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:23:58    at java.base@21.0.1-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
20:23:58  
20:23:58  "Concurrent Mark Helper" prio=1 Id=14 RUNNABLE
20:23:58  
20:23:58  "Finalizer thread" prio=5 Id=15 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=16 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=17 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=18 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=19 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=20 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=21 RUNNABLE
20:23:58  
20:23:58  "GC Worker" prio=5 Id=22 RUNNABLE
20:23:58  
20:23:58  "Attach API wait loop" prio=10 Id=25 RUNNABLE
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.IPC.waitSemaphore(Native Method)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.CommonDirectory.waitSemaphore(CommonDirectory.java:264)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.WaitLoop.waitForNotification(WaitLoop.java:66)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.WaitLoop.run(WaitLoop.java:157)
20:23:58  
20:23:58  "pool-1-thread-1" prio=5 Id=26 WAITING
20:23:58    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
20:23:58    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:23:58  
20:23:58  "ForkJoinPool-1-worker-1" prio=5 Id=29 TIMED_WAITING
20:23:58    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:449)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1891)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1809)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
20:23:58  
20:23:58  "VirtualThread-unparker" prio=5 Id=30 WAITING
20:23:58    at java.base@21.0.1-internal/jdk.internal.misc.Unsafe.park(Native Method)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
20:23:58    at java.base@21.0.1-internal/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
20:23:58    at java.base@21.0.1-internal/java.lang.Thread.run(Thread.java:1595)
20:23:58    at java.base@21.0.1-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
20:23:58  
20:23:58  "file lock watchdog" prio=10 Id=32 TIMED_WAITING
20:23:58    at java.base@21.0.1-internal/java.lang.Object.waitImpl(Native Method)
20:23:58    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:255)
20:23:58    at java.base@21.0.1-internal/java.lang.Object.wait(Object.java:221)
20:23:58    at java.base@21.0.1-internal/java.util.TimerThread.mainLoop(Timer.java:570)
20:23:58    at java.base@21.0.1-internal/java.util.TimerThread.run(Timer.java:523)
20:23:58  
20:23:58  "Attachment portNumber: 56931" prio=10 Id=34 RUNNABLE
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.dumpAllThreadsImpl(Native Method)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.getThreadInfo(DiagnosticUtils.java:249)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.DiagnosticUtils.executeDiagnosticCommand(DiagnosticUtils.java:185)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.Attachment.doCommand(Attachment.java:248)
20:23:58    at java.base@21.0.1-internal/openj9.internal.tools.attach.target.Attachment.run(Attachment.java:159)
20:23:58  
20:23:58  
20:23:58  --- Timeout information end.

@fengxue-IS @TobiAjila did we do something to regress?

fengxue-IS commented 10 months ago

I am not aware of any vthread related PR that could affect this test in the last 2 days

fengxue-IS commented 10 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_0/90/

@fengxue-IS @TobiAjila did we do something to regress?

I tried running test in personal grinder, all passed @pshipton can you try launching a grinder on https://openj9-jenkins.osuosl.org/computer/win2019-x64-3/ machine as both failure occurred on this machine, previous builds that passed were using a different machine.

pshipton commented 10 months ago

Started https://openj9-jenkins.osuosl.org/job/Grinder/3047/ on just the failed test. The nightly builds have started, it will take some time to get the machine. If this doesn't fail I can try running all jdk_lang_0, sometimes it's necessary to run the entire suite to reproduce a failure.

pshipton commented 10 months ago

Failed on iteration 2. These newer failures seem different from the original https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1798861359 in that they contain sun.nio.ch.Net.poll which is maybe a machine issue.

pshipton commented 10 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_windows_Nightly_testList_1/96/ jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0 timeout

pshipton commented 10 months ago

Skynet workaround ready for plinux using -Xiss4K -Xssi2K

@fengxue-IS Since the -Xiss default is 2K, it should be sufficient to just use -Xssi2K instead of the 16K default?

fengxue-IS commented 9 months ago

AIX still fails after including fix from https://github.com/fengxue-IS/openj9/commit/adeb21ef16f600ec70dbde31d21ad9e1d3642e02 and https://github.com/fengxue-IS/openj9-openjdk-jdk21/commit/0e33fa8f05b856de91a0a26341c3af12b4bbc220, investigating root cause of issue. At the same time, I am testing with a smaller test set of 100000 total continuation to see if test pass consistently on AIX if memory pressure is reduced.