search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

jdk19 OpenJDK java/lang/Thread/virtual/stress/Skynet.java hang / timeout #16728

Closed pshipton closed 10 months ago

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/111/ jdk_lang_0 java/lang/Thread/virtual/stress/Skynet.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/111/openjdk_test_output.tar.gz

21:55:35  ACTION: main -- Error. Program `/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 2400000ms, elapsed time including timeout handling was 2404113ms).
21:55:35  REASON: User specified action: run main/othervm/timeout=300 --enable-preview Skynet 
21:55:35  TIME:   2404.116 seconds
21:55:35  messages:
21:55:35  command: main --enable-preview Skynet
21:55:35  reason: User specified action: run main/othervm/timeout=300 --enable-preview Skynet 
21:55:35  Mode: othervm [/othervm specified]
21:55:35  Timeout information:
21:55:35  Running jcmd on process 2529589
21:55:35  Dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/aqa-tests/TKG/output_1676426825596/jdk_lang_0/work/scratch/2/core.20230214.215530.2529589.0001.dmp
21:55:35  Dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/aqa-tests/TKG/output_1676426825596/jdk_lang_0/work/scratch/2/javacore.20230214.215531.2529589.0002.txt
21:55:35  Running jstack on process 2529589
21:55:35  2023-02-14T21:55:32.813679073
21:55:35  Virtual machine: 2529589 JVM information:
21:55:35  JRE 19 Linux aarch64-64-Bit Compressed References 20230215_180 (JIT enabled, AOT enabled)
21:55:35  OpenJ9   - 711ca6c2adb
21:55:35  OMR      - 6adea0343aa
21:55:35  JCL      - 47925e78c4e based on jdk-19.0.2+7
21:55:35  
21:55:35  "main" prio=5 Id=2 WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.join(Thread.java:2151)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.join(Thread.java:2227)
21:55:35    at app//com.sun.javatest.regtest.agent.MainWrapper.main(MainWrapper.java:74)
21:55:35  
21:55:35  "JIT Compilation Thread-000" prio=10 Id=4 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-001 Suspended" prio=10 Id=5 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-002 Suspended" prio=10 Id=6 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-003 Suspended" prio=10 Id=7 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-004 Suspended" prio=10 Id=8 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-005 Suspended" prio=10 Id=9 RUNNABLE
21:55:35  
21:55:35  "JIT Compilation Thread-006 Suspended" prio=10 Id=10 RUNNABLE
21:55:35  
21:55:35  "JIT Diagnostic Compilation Thread-007 Suspended" prio=10 Id=11 RUNNABLE
21:55:35  
21:55:35  "JIT-SamplerThread" prio=10 Id=12 TIMED_WAITING
21:55:35  
21:55:35  "IProfiler" prio=5 Id=13 RUNNABLE
21:55:35  
21:55:35  "Common-Cleaner" prio=8 Id=3 TIMED_WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:137)
21:55:35    at java.base@19.0.2-internal/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.run(Thread.java:1573)
21:55:35    at java.base@19.0.2-internal/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)
21:55:35  
21:55:35  "Concurrent Mark Helper" prio=1 Id=14 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=15 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=16 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=17 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=18 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=19 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=20 RUNNABLE
21:55:35  
21:55:35  "GC Worker" prio=5 Id=21 RUNNABLE
21:55:35  
21:55:35  "Attach API wait loop" prio=10 Id=24 RUNNABLE
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.IPC.waitSemaphore(Native Method)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.CommonDirectory.waitSemaphore(CommonDirectory.java:264)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.WaitLoop.waitForNotification(WaitLoop.java:66)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.WaitLoop.run(WaitLoop.java:157)
21:55:35  
21:55:35  "Finalizer thread" prio=5 Id=25 RUNNABLE
21:55:35  
21:55:35  "MainThread" prio=5 Id=27 WAITING
21:55:35    at java.base@19.0.2-internal/jdk.internal.misc.Unsafe.park(Native Method)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue$TransferStack$SNode.block(SynchronousQueue.java:288)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:397)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:886)
21:55:35    at app//Skynet$Channel.receive(Skynet.java:110)
21:55:35    at app//Skynet.skynet(Skynet.java:60)
21:55:35    at app//Skynet.main(Skynet.java:50)
21:55:35    at java.base@19.0.2-internal/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
21:55:35    at java.base@19.0.2-internal/java.lang.reflect.Method.invoke(Method.java:578)
21:55:35    at app//com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
21:55:35    at java.base@19.0.2-internal/java.lang.Thread.run(Thread.java:1573)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-2" prio=5 Id=31 RUNNABLE
21:55:35    at java.base@19.0.2-internal/jdk.internal.vm.Continuation.enterImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/jdk.internal.vm.Continuation.run(Continuation.java:198)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:218)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-3" prio=5 Id=41 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiUnmountEnd(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.afterYield(VirtualThread.java:396)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:223)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-6" prio=5 Id=47 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiUnmountEnd(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.afterTerminate(VirtualThread.java:424)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:221)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "ForkJoinPool-1-worker-7" prio=5 Id=48 RUNNABLE
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.notifyJvmtiMountBegin(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.VirtualThread.runContinuation(VirtualThread.java:215)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
21:55:35    at java.base@19.0.2-internal/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
21:55:35  
21:55:35  "file lock watchdog" prio=10 Id=10700077 TIMED_WAITING
21:55:35    at java.base@19.0.2-internal/java.lang.Object.waitImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:255)
21:55:35    at java.base@19.0.2-internal/java.lang.Object.wait(Object.java:221)
21:55:35    at java.base@19.0.2-internal/java.util.TimerThread.mainLoop(Timer.java:569)
21:55:35    at java.base@19.0.2-internal/java.util.TimerThread.run(Timer.java:522)
21:55:35  
21:55:35  "Attachment portNumber: 40951" prio=10 Id=10708739 RUNNABLE
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.dumpAllThreadsImpl(Native Method)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.getThreadInfo(DiagnosticUtils.java:245)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.DiagnosticUtils.executeDiagnosticCommand(DiagnosticUtils.java:181)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.Attachment.doCommand(Attachment.java:248)
21:55:35    at java.base@19.0.2-internal/openj9.internal.tools.attach.target.Attachment.run(Attachment.java:159)
21:55:35  
21:55:35  
21:55:35  --- Timeout information end.
21:55:35  elapsed time (seconds): 2404.116
21:55:35  configuration:
21:55:35  STDOUT:
21:55:35  Result: 499999500000 in 396129 ms
21:55:35  Result: 499999500000 in 141690 ms
21:55:35  Result: 499999500000 in 344331 ms
21:55:35  Result: 499999500000 in 326083 ms
21:55:35  Result: 499999500000 in 300962 ms
21:55:35  Result: 499999500000 in 379486 ms
21:55:35  Result: 499999500000 in 222160 ms
21:55:35  Result: 499999500000 in 194157 ms
21:55:35  Result: 499999500000 in 50598 ms
21:55:35  Timeout refired 2400 times
pshipton commented 11 months ago

@fengxue-IS I assume we want these changes merged regardless? If so we should start the PR/review process and get them in so that testing for all platforms is running with them sooner.

pshipton commented 11 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/106/ jdk_lang_0 java/lang/Thread/virtual/stress/Skynet.java#default TEST RESULT: Failed. Unexpected exit from test [exit code: 137] https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/106/openjdk_test_output.tar.gz

fengxue-IS commented 11 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/106/ jdk_lang_0 java/lang/Thread/virtual/stress/Skynet.java#default TEST RESULT: Failed. Unexpected exit from test [exit code: 137] https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/106/openjdk_test_output.tar.gz

I don't see anything wrong in the test output, exit code 137 is usually from OS killing the process due to insufficient memory. which I suspect is related to machine. Launched 50x grinder to reproduce issue

fengxue-IS commented 11 months ago

50x grinder passed, I will try reruning jdk_lang_0 to see if failure only reproduces with entire suite

10x Grinder on jdk_lang_0

babsingh commented 11 months ago

Update (below failures are only seen on AIX)

LinHu2016 commented 11 months ago

Update: [issue 2] in https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1843799810 the issue also happens on Balanced GC and gencon without concurrent mark, it looks not related with concurrent GC. and it looks separate issue with issue3.

[issue 3] I am majorly focus on debugging this one, it looks only related with concurrent GC, I have seen wired continuation state for the crash cases, it is trying to unmount the continuation, which is concurrently scaned by GC, logically we should never see this happen, because we have synchronization between mount and concurrent scan and atomic access for the state for handling race condition, if concurrent scan comes first, the mounting would be blocked until concurrent scan is finished, if mounting comes first, it would skip concurrent scan(we don't need to scan mounted continuation). looks there is a hole for the synchronization, still tracing the root of cause. and in similar case if the umount is last unmount, it potentially could trigger the assertion in issue 1, and currently the PR for issue 1 would not cover the case.

tajila commented 11 months ago

[issue 2] in https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1843799810 the issue also happens on Balanced GC and gencon without concurrent mark, it looks not related with concurrent GC. and it looks separate issue with issue3.

The crash occurs while a virtual thread is mounted in the critical region. 

> !j9object 0x0A000000068BD3E0
!J9Object 0x0A000000068BD3E0 {
        struct J9Class* clazz = !j9class 0x10021C01D00 // java/lang/VirtualThread
        Object flags = 0x00000000;
        J lockword = 0x0000000000000008 (offset = 0) (java/lang/Object) <hidden>
        J eetop = 0x0000000000000000 (offset = 8) (java/lang/Thread)
        J tid = 0x00000000007B304C (offset = 16) (java/lang/Thread)
        Ljava/lang/String; name = !fj9object 0xa0000000000ae18 (offset = 40) (java/lang/Thread)
        Z interrupted = 0x00000000 (offset = 152) (java/lang/Thread)
        Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0xa000000000583c0 (offset = 48) (java/lang/Thread)
        Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0xa00000000058c08 (offset = 56) (java/lang/Thread)
        Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 64) (java/lang/Thread)
        Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 72) (java/lang/Thread)
        Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 80) (java/lang/Thread)
        Ljava/lang/Object; scopedValueBindings = !fj9object 0xa000000000047e8 (offset = 88) (java/lang/Thread)
        Ljava/lang/Object; interruptLock = !fj9object 0xa000000068bd4d0 (offset = 96) (java/lang/Thread)
        Ljava/lang/Object; parkBlocker = !fj9object 0xa0000000394c950 (offset = 104) (java/lang/Thread)
        Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 112) (java/lang/Thread)
        Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 120) (java/lang/Thread)
        Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 128) (java/lang/Thread)
        J threadLocalRandomSeed = 0x0000000000000000 (offset = 24) (java/lang/Thread)
        I threadLocalRandomProbe = 0x00000000 (offset = 156) (java/lang/Thread)
        I threadLocalRandomSecondarySeed = 0x00000000 (offset = 160) (java/lang/Thread)
        Ljdk/internal/vm/ThreadContainer; container = !fj9object 0xa00000000117520 (offset = 136) (java/lang/Thread)
        Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 144) (java/lang/Thread)
        Z started = 0x00000000 (offset = 164) (java/lang/Thread)
        J tls = 0x0000000000000000 (offset = 32) (java/lang/Thread) <hidden>
        I isSuspendedInternal = 0x00000000 (offset = 168) (java/lang/Thread) <hidden>
        Ljava/util/concurrent/Executor; scheduler = !fj9object 0xa00000000160448 (offset = 184) (java/lang/VirtualThread)
        Ljdk/internal/vm/Continuation; cont = !fj9object 0xa000000068bd4e0 (offset = 192) (java/lang/VirtualThread)
        Ljava/lang/Runnable; runContinuation = !fj9object 0xa000000068bd538 (offset = 200) (java/lang/VirtualThread)
        I state = 0x00000004 (offset = 172) (java/lang/VirtualThread)
        Z parkPermit = 0x00000000 (offset = 224) (java/lang/VirtualThread)
        Ljava/lang/Thread; carrierThread = !fj9object 0xa0000000019d1f0 (offset = 208) (java/lang/VirtualThread)
        Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x0 (offset = 216) (java/lang/VirtualThread)
        J inspectorCount = 0xFFFFFFFFFFFFFFFF (offset = 176) (java/lang/VirtualThread) <hidden>

However, it seems like the mount did not complete successfully as vmthread->currentContinuation is NULL.

Also, it looks like the stack of the carrier thread is also messed up because it only shows a single native frame which would only be the case at thread start.

I noticed a bit of corruption of the vmthread as well

> !j9vmthread 0x10023898e00
J9VMThread at 0x10023898e00 {
  Fields for J9VMThread:
        0x0: struct JNINativeInterface_* functions = !jninativeinterface_ 0x000001002100EB18
        0x8: struct J9JavaVM* javaVM = !j9javavm 0x00000000000068E2
        0x10: UDATA* arg0EA = !j9x 0x0A0000000019D2C0
        0x18: UDATA* bytecodes = !j9x 0x0A000000036853E8
        0x20: UDATA* sp = !j9x 0x0000000000000003
        0x28: U8* pc = !j9x 0x0000000000000000
        0x30: struct J9Method* literals = !j9method 0x0000000000000000
        0x38: UDATA jitStackFrameFlags = 0x0A00000000EA78B8 (720575940394645688)
        0x40: struct J9Object* jitException = !j9object 0x0000000000000000
        0x48: struct J9Object* currentException = !j9object 0x0000000000000000
        0x50: UDATA* stackOverflowMark = !j9x 0x0000000000000000
        0x58: UDATA* stackOverflowMark2 = !j9x 0x0000000000000000
        0x60: U8* heapAlloc = !j9x 0x0000000000000000
        0x68: U8* heapTop = !j9x 0x0A00000000EA78B8 // ""
        0x70: IDATA tlhPrefetchFTA = 0x0A00000000EA79B8 (720575940394645944)
        0x78: U8* nonZeroHeapAlloc = !j9x 0x0A00000000EA79B8 // ""
        0x80: U8* nonZeroHeapTop = !j9x 0x0000000042000244
        0x88: IDATA nonZeroTlhPrefetchFTA = 0x0000010011269B04 (1099799370500)
        0x90: struct J9ThreadMonitor* publicFlagsMutex = !j9threadmonitor 0x00000100357A7E81
        0x98: UDATA publicFlags = 0x00000100357A7EA0 (1100408848032)
        0xa0: UDATA compressObjectReferences = 0x0000000000000000 (0)
        0xa8: struct J9Object* threadObject = !j9object 0x0900000008202890<FAULT>
        0xb0: void* lowTenureAddress = !j9x 0x0000000000000000
        0xb8: void* highTenureAddress = !j9x 0x0000000000000000
        0xc0: void* heapBaseForActiveCardTable = !j9x 0xB76543213CB68619
        0xc8: void* activeCardTableBase = !j9x 0x0000000000000190
        0xd0: UDATA heapSizeForActiveCardTable = 0x0900000008202210 (648518346477675024)
        0xd8: void* heapBaseForBarrierRange0 = !j9x 0x09001000A07A7978
        0xe0: UDATA heapSizeForBarrierRange0 = 0x5B5B5B5B00000001 (6582955726732263425)
        0xe8: UDATA* jniLocalReferences = !j9x 0x00000000000001D0
        0xf0: UDATA tempSlot = 0xB12345673AF0806F (-5682621995138645905)
        0xf8: void* jitReturnAddress = !j9x 0x0000000000000190
        0x100: void* floatTemp1 = !j9x 0x0900000008202210
        0x108: void* floatTemp2 = !j9x 0x09001000A07A7978
        0x110: void* floatTemp3 = !j9x 0x000001005DEC06B0
        0x118: void* floatTemp4 = !j9x 0x0000000000000000
        0x120: UDATA returnValue = 0x000001005DEC0688 (1101087377032)
        0x128: UDATA returnValue2 = 0x0000000000000003 (3)
        0x130: UDATA* objectFlagSpinLockAddress = !j9x 0x0000000000000000
        0x138: struct J9JavaStack* stackObject = !j9javastack 0x000001005DEBFC78
        0x140: struct J9Thread* osThread = !j9thread 0x000001005DEBFC78

osThread and stackObject have the same value

hzongaro commented 11 months ago

[ISSUE 2] There is also a crash in runJavaThread, which goes away with -Xint (@LinHu2016 noticed the same). Grinder with -Xint (no runJavaThread crash seen)

@tajila, I'm not sure whether your comment earlier today implied anything about whether Issue 2 could still be JIT-related. In any event, I was trying to narrow things down, as I couldn't spot anything JIT-related of interest. I ended up being able to reproduce the runJavaThread crash in Internal grinder run #36800 running with -XX:+MergeCompilerOptions -Xjit:verbose,vlog=vlog.txt,limit={java/util/xyzzy*} -Xnoaot which results in no methods being compiled.

I'm running again with -Xint, just in case just having the JIT initialized is somehow causing problems.

hzongaro commented 11 months ago

No luck reproducing the crash in runJavaThread with -Xint in internal grinder run 36806, but was able to reproduce it again in internal grinder run 36805 with -Xjit:verbose,vlog=vlog.txt,exclude={*} -Xnoaot - that is, compiling no methods.

tajila commented 11 months ago

@tajila, I'm not sure whether your comment earlier today implied anything about whether Issue 2 could still be JIT-related.

At this point I think its unlikely. It looks like the continuation was freed when it shouldn't have. Perhaps @LinHu2016 has an update.

tajila commented 11 months ago

Just to elaborate some more. It looks like what happens is the Virtual thread is in the process of being unmounted (yield is called). We then swap the continuation state to the carrier thread state. The issue is that the carrier thread state has been cleared, which leads me to believe that the GC has freed it when it was not supposed to.

LinHu2016 commented 11 months ago

one small hole in synchronization between continuation mount and concurrent scan has been found and fixed, "the hole" could potentially cause wrong continuation state or concurrent GC conflict with continuation swapping on all of platform, but we could not reproduce the issue on others platforms than AIX, with the fix, we still experience the crashes, but the failure rate has been reduced from 40% to 20%, and the fix only help in concurrent gc case. There is another issue, which might be related with compiler or compiler optimization, we have found some checking menthods with local variable return inconsistent result or wrong result, it looks related with most of crash cases, for the crashes in runJavaThread still not very clear if it is also related with the "compiler problem".

LinHu2016 commented 11 months ago

Update: 1, looks like there is compiling/running issue with logic "do {} while for the infinity loop" on power machines, we were using do { if (condition) { ... continute; } }while {false};for the infinity loop with a break to handle waiting concurrent gc finish during mounting, but compiler did not generate correct logic( the generated logic never loop, we also tried to update with do { if (condition) {...}else {break;}while (true); , it also has similar issue.) , but for(;;) { if (condition) {...} else{break;}} works fine, it could cause wrong state and concurrent gc scan conflict with continuation swapping, with for loop update, we pass the test 50/50 on powerpc le linux and no issue except the crash in runJavaThread on aix.

the crashes in runJavaThread is sperated issue, it looks not directly related with GC. it looks like AIX only issue, we never reproduce it in Linux PPC le it happens on all of gc policies (gen, balanced, optthruput...) it happens on concurrent GC disabled mode it can also be reproduced with -Xgcpolicy:nogc

LinHu2016 commented 11 months ago 
[2023-12-15T15:15:38.022Z] == ITERATION 1 ==
[2023-12-15T15:15:38.022Z] 
[2023-12-15T15:15:38.022Z] TEST SETUP:
[2023-12-15T15:15:38.364Z] 
[2023-12-15T15:15:38.364Z] Attempting to destroy all caches in cacheDir /home/jenkins/.cache/javasharedresources/
[2023-12-15T15:15:38.364Z] 
[2023-12-15T15:15:38.364Z] JVMSHRC806I Compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
[2023-12-15T15:15:38.364Z] JVMSHRC806I Compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
[2023-12-15T15:15:38.364Z] JVMSHRC807I Non-compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xcompressedrefs if you want to destroy a compressed references cache.
[2023-12-15T15:15:38.364Z] JVMSHRC005I No shared class caches available
[2023-12-15T15:15:38.364Z] cache cleanup done
[2023-12-15T15:15:38.364Z] 
[2023-12-15T15:15:38.364Z] TESTING:
[2023-12-15T15:15:40.090Z] Directory "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17026533377871/jdk_custom_0_ITER_1/work" not found: creating
[2023-12-15T15:15:40.090Z] Directory "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17026533377871/jdk_custom_0_ITER_1/report" not found: creating
[2023-12-15T15:15:49.299Z] XML output with verification to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work
[2023-12-15T15:16:12.847Z] --------------------------------------------------
[2023-12-15T15:16:12.847Z] TEST: java/lang/Thread/virtual/stress/Skynet.java#default
[2023-12-15T15:16:12.847Z] TEST JDK: /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image
[2023-12-15T15:16:12.847Z] 
[2023-12-15T15:16:12.847Z] ACTION: build -- Passed. Build successful
[2023-12-15T15:16:12.847Z] REASON: Named class compiled on demand
[2023-12-15T15:16:12.847Z] TIME:   2.738 seconds
[2023-12-15T15:16:12.847Z] messages:
[2023-12-15T15:16:12.847Z] command: build Skynet
[2023-12-15T15:16:12.847Z] reason: Named class compiled on demand
[2023-12-15T15:16:12.847Z] started: Fri Dec 15 10:15:49 EST 2023
[2023-12-15T15:16:12.847Z] Test directory:
[2023-12-15T15:16:12.847Z]   compile: Skynet
[2023-12-15T15:16:12.847Z] finished: Fri Dec 15 10:15:52 EST 2023
[2023-12-15T15:16:12.847Z] elapsed time (seconds): 2.738
[2023-12-15T15:16:12.847Z] 
[2023-12-15T15:16:12.847Z] ACTION: compile -- Passed. Compilation successful
[2023-12-15T15:16:12.847Z] REASON: .class file out of date or does not exist
[2023-12-15T15:16:12.847Z] TIME:   2.725 seconds
[2023-12-15T15:16:12.847Z] messages:
[2023-12-15T15:16:12.847Z] command: compile /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress/Skynet.java
[2023-12-15T15:16:12.847Z] reason: .class file out of date or does not exist
[2023-12-15T15:16:12.847Z] started: Fri Dec 15 10:15:49 EST 2023
[2023-12-15T15:16:12.847Z] Mode: agentvm
[2023-12-15T15:16:12.847Z] Agent id: 1
[2023-12-15T15:16:12.847Z] finished: Fri Dec 15 10:15:52 EST 2023
[2023-12-15T15:16:12.847Z] elapsed time (seconds): 2.725
[2023-12-15T15:16:12.847Z] configuration:
[2023-12-15T15:16:12.847Z] Boot Layer (javac runtime environment)
[2023-12-15T15:16:12.847Z]   class path: /home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/javatest.jar 
[2023-12-15T15:16:12.847Z]               /home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/jtreg.jar 
[2023-12-15T15:16:12.847Z]   patch:      java.base /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/patches/java.base
[2023-12-15T15:16:12.847Z] 
[2023-12-15T15:16:12.847Z] javac compilation environment
[2023-12-15T15:16:12.848Z]   source path: /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress
[2023-12-15T15:16:12.848Z]   class path:  /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress
[2023-12-15T15:16:12.848Z]                /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d
[2023-12-15T15:16:12.848Z] 
[2023-12-15T15:16:12.848Z] rerun:
[2023-12-15T15:16:12.848Z] cd /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch && \
[2023-12-15T15:16:12.848Z] DISPLAY=unix:0 \
[2023-12-15T15:16:12.848Z] HOME=/home/jenkins \
[2023-12-15T15:16:12.848Z] LANG=en_US \
[2023-12-15T15:16:12.848Z] PATH=/bin:/usr/bin:/usr/sbin \
[2023-12-15T15:16:12.848Z] TZ=America/New_York \
[2023-12-15T15:16:12.848Z]     /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/javac \
[2023-12-15T15:16:12.848Z]         -J-ea \
[2023-12-15T15:16:12.848Z]         -J-esa \
[2023-12-15T15:16:12.848Z]         -J-Xmx512m \
[2023-12-15T15:16:12.848Z]         -J-Xverbosegclog \
[2023-12-15T15:16:12.848Z]         -J-Xgcpolicy:nogc \
[2023-12-15T15:16:12.848Z]         -J-Xmx2g \
[2023-12-15T15:16:12.848Z]         -J-Dtest.vm.opts='-ea -esa -Xmx512m -Xverbosegclog -Xgcpolicy:nogc -Xmx2g' \
[2023-12-15T15:16:12.848Z]         -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-Xverbosegclog -J-Xgcpolicy:nogc -J-Xmx2g' \
[2023-12-15T15:16:12.848Z]         -J-Dtest.compiler.opts= \
[2023-12-15T15:16:12.848Z]         -J-Dtest.java.opts= \
[2023-12-15T15:16:12.848Z]         -J-Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
[2023-12-15T15:16:12.848Z]         -J-Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
[2023-12-15T15:16:12.848Z]         -J-Dtest.timeout.factor=8.0 \
[2023-12-15T15:16:12.848Z]         -J-Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \
[2023-12-15T15:16:12.848Z]         -J-Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \
[2023-12-15T15:16:12.848Z]         -J-Dtest.name=java/lang/Thread/virtual/stress/Skynet.java#default \
[2023-12-15T15:16:12.848Z]         -J-Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress/Skynet.java \
[2023-12-15T15:16:12.848Z]         -J-Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.848Z]         -J-Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.848Z]         -J-Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d \
[2023-12-15T15:16:12.848Z]         -J-Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d \
[2023-12-15T15:16:12.848Z]         -J-Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.848Z]         -d /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d \
[2023-12-15T15:16:12.848Z]         -sourcepath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.848Z]         -classpath /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d /home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress/Skynet.java
[2023-12-15T15:16:12.848Z] 
[2023-12-15T15:16:12.848Z] ACTION: main -- Failed. Unexpected exit from test [exit code: 255]
[2023-12-15T15:16:12.848Z] REASON: User specified action: run main/othervm/timeout=300 -Xmx1g -Xssi2K -Xnocompressedrefs Skynet 
[2023-12-15T15:16:12.848Z] TIME:   16.51 seconds
[2023-12-15T15:16:12.848Z] messages:
[2023-12-15T15:16:12.848Z] command: main -Xmx1g -Xssi2K -Xnocompressedrefs Skynet
[2023-12-15T15:16:12.848Z] reason: User specified action: run main/othervm/timeout=300 -Xmx1g -Xssi2K -Xnocompressedrefs Skynet 
[2023-12-15T15:16:12.848Z] started: Fri Dec 15 10:15:52 EST 2023
[2023-12-15T15:16:12.848Z] Mode: othervm [/othervm specified]
[2023-12-15T15:16:12.848Z] finished: Fri Dec 15 10:16:09 EST 2023
[2023-12-15T15:16:12.848Z] elapsed time (seconds): 16.51
[2023-12-15T15:16:12.848Z] configuration:
[2023-12-15T15:16:12.848Z] STDOUT:
[2023-12-15T15:16:12.848Z] STDERR:
[2023-12-15T15:16:12.848Z] Unhandled exception
[2023-12-15T15:16:12.848Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.848Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.848Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.848Z] R0=0000000000000001 R1=0000010023469660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.848Z] R4=0000010022D90A00 R5=0000010010FD4C80 R6=0000010024842360 R7=0000000000000088
[2023-12-15T15:16:12.848Z] R8=FFFFFFFFFFFFFFFF R9=00000100222ED238 R10=0A000200097166D5 R11=0000000000000001
[2023-12-15T15:16:12.848Z] R12=0000000042000041 R13=0000010023474800 R14=0000010024842380 R15=0000010022D90A00
[2023-12-15T15:16:12.848Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=000000000000284C
[2023-12-15T15:16:12.848Z] R20=0A0000000025A850 R21=000000000000284C R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.848Z] R24=0A000000012B6048 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.848Z] R28=0A00000001259FC0 R29=0A0000000025A850 R30=0000000000008020 R31=0000010021B6F900
[2023-12-15T15:16:12.848Z] IAR=0000000000000000 LR=0000000000000001 MSR=A00000000200D032 CTR=0000010010FD4C80
[2023-12-15T15:16:12.848Z] CR=4200004100000001 FPSCR=8202000000000000 XER=0000000182020000
[2023-12-15T15:16:12.848Z] Unhandled exception
[2023-12-15T15:16:12.848Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.848Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.848Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.848Z] R0=0A0000000537B0B8 R1=00000100231A9660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.848Z] R4=09001000A1565D88 R5=0000010022C38D00 R6=0000000000000003 R7=0A00000001517030
[2023-12-15T15:16:12.848Z] R8=0000010022C38D00 R9=0000010021B98A00 R10=00000100231A98A0 R11=000001002B89F8D8
[2023-12-15T15:16:12.848Z] R12=0000000000000000 R13=00000100231B4800 R14=000001002B8A01A0 R15=0000010022C38D00
[2023-12-15T15:16:12.848Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=09001000A0E6D368
[2023-12-15T15:16:12.848Z] R20=0000000000000000 R21=0000000000000000 R22=0A000000013C1F18 R23=0A0000000537B018
[2023-12-15T15:16:12.848Z] R24=0A0000000537B0A0 R25=0000000000000001 R26=0000000000000000 R27=0000000000000000
[2023-12-15T15:16:12.848Z] R28=0A000000013C0C48 R29=0A0000000537B0B8 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.848Z] IAR=0A0000000537B0B8 LR=0A0000000537B0B8 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.848Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.848Z] Unhandled exception
[2023-12-15T15:16:12.848Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.848Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.848Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.848Z] R0=0A00000001908988 R1=0000010022EE9660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.848Z] R4=09001000A1565D88 R5=0000010022320100 R6=0000000000000003 R7=0A000000006DE730
[2023-12-15T15:16:12.848Z] R8=0000010022320100 R9=0000010021B98A00 R10=0000010022EE98A0 R11=0000010024852110
[2023-12-15T15:16:12.848Z] R12=0000000000000000 R13=0000010022EF4800 R14=00000100248524C0 R15=0000010022320100
[2023-12-15T15:16:12.848Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00000000000028E7
[2023-12-15T15:16:12.849Z] R20=0A0000000025A850 R21=00000000000028E7 R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.849Z] R24=0A000000013FA8F8 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.849Z] R28=0A00000001259FC0 R29=0A0000000025A850 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.849Z] IAR=0A00000001908988 LR=0A00000001908988 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.849Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.849Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.849Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.849Z] FPR2 3fc719ead72ce474 (f: 3610043392.000000, d: 1.804784e-01)
[2023-12-15T15:16:12.849Z] FPR3 3f7ae90000000000 (f: 0.000000, d: 6.569862e-03)
[2023-12-15T15:16:12.849Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.849Z] FPR5 40a6820000000000 (f: 0.000000, d: 2.881000e+03)
[2023-12-15T15:16:12.849Z] FPR6 4070b2623764c12d (f: 929349952.000000, d: 2.671490e+02)
[2023-12-15T15:16:12.849Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.849Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.849Z] FPR9 403187837ecdc890 (f: 2127415424.000000, d: 1.752935e+01)
[2023-12-15T15:16:12.849Z] FPR10 c024000000000000 (f: 0.000000, d: -1.000000e+01)
[2023-12-15T15:16:12.849Z] FPR11 c30c6bf526340000 (f: 640942080.000000, d: -1.000000e+15)
[2023-12-15T15:16:12.849Z] FPR12 b9b4484bfeebc2a0 (f: 4276863744.000000, d: -1.000000e-30)
[2023-12-15T15:16:12.849Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] Unhandled exception
[2023-12-15T15:16:12.849Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.849Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.849Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.849Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.849Z] R0=0A000000059D4368 R1=00000100230EC660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.849Z] R4=09001000A1565D88 R5=000001002251F700 R6=0000000000000003 R7=0A00000001C186A8
[2023-12-15T15:16:12.849Z] R8=000001002251F700 R9=0000010021B98A00 R10=00000100230EC8A0 R11=000001002C7339D8
[2023-12-15T15:16:12.849Z] R12=0000000000000000 R13=00000100230F7800 R14=000001002C7342A0 R15=000001002251F700
[2023-12-15T15:16:12.849Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=09001000A0E6D368
[2023-12-15T15:16:12.849Z] R20=0000000000000000 R21=0000000000000000 R22=0A00000001C185A8 R23=0A000000059D4318
[2023-12-15T15:16:12.849Z] R24=0A000000059D4350 R25=0000000000000001 R26=0000000000000000 R27=0000000000000000
[2023-12-15T15:16:12.849Z] R28=0A00000001AAA208 R29=0A000000059D4368 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.849Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.849Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.849Z] IAR=0A000000059D4368 LR=0A000000059D4368 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.849Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.849Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.849Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.849Z] FPR2 3fc706e042ed4c65 (f: 1122847872.000000, d: 1.798973e-01)
[2023-12-15T15:16:12.849Z] FPR3 3f8ba88000000000 (f: 0.000000, d: 1.350498e-02)
[2023-12-15T15:16:12.849Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.849Z] FPR5 40a6820000000000 (f: 0.000000, d: 2.881000e+03)
[2023-12-15T15:16:12.849Z] FPR6 4070b2623764c12d (f: 929349952.000000, d: 2.671490e+02)
[2023-12-15T15:16:12.849Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.849Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.849Z] FPR9 403187837ecdc890 (f: 2127415424.000000, d: 1.752935e+01)
[2023-12-15T15:16:12.849Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.849Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.849Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.849Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.849Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.849Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.849Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.849Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.849Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.849Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.849Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.849Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.849Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.849Z] ---------------------------------------
[2023-12-15T15:16:12.850Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.850Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.850Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.850Z] FPR2 3fc6e0876697e4d4 (f: 1721230592.000000, d: 1.787271e-01)
[2023-12-15T15:16:12.850Z] FPR3 3fa1e80800000000 (f: 0.000000, d: 3.497338e-02)
[2023-12-15T15:16:12.850Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.850Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.850Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.850Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.850Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.850Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.850Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.850Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.850Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.850Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.850Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.850Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.850Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.850Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.850Z] ---------------------------------------
[2023-12-15T15:16:12.850Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.850Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.850Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.850Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.850Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.850Z] FPR2 3fc6ea9418641fc0 (f: 409214912.000000, d: 1.790338e-01)
[2023-12-15T15:16:12.850Z] FPR3 3f9d0f1000000000 (f: 0.000000, d: 2.837777e-02)
[2023-12-15T15:16:12.850Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.850Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.850Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.850Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.850Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.850Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.850Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.850Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.850Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.850Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.850Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.850Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.850Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.850Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.850Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.850Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.850Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.850Z] Unhandled exception
[2023-12-15T15:16:12.850Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.850Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.850Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.850Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.850Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.850Z] ---------------------------------------
[2023-12-15T15:16:12.850Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.850Z] R0=0A00000005A6AE50 R1=00000100233B9660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.850Z] R4=09001000A1565D88 R5=0000010022AD2E00 R6=0000000000000003 R7=0A00000001C29F80
[2023-12-15T15:16:12.850Z] R8=0000010022AD2E00 R9=0000010021B98A00 R10=00000100233B98A0 R11=000001002C7BD8D8
[2023-12-15T15:16:12.850Z] R12=0000000000000000 R13=00000100233C4800 R14=000001002C7BE1A0 R15=0000010022AD2E00
[2023-12-15T15:16:12.850Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=09001000A0E6D368
[2023-12-15T15:16:12.850Z] R20=0000000000000000 R21=0000000000000000 R22=0A00000001C29E80 R23=0A00000005A6AE00
[2023-12-15T15:16:12.850Z] R24=0A00000005A6AE38 R25=0000000000000001 R26=0000000000000000 R27=0000000000000000
[2023-12-15T15:16:12.850Z] R28=0A00000001C291E0 R29=0A00000005A6AE50 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.850Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.850Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.851Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.851Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.851Z] IAR=0A00000005A6AE50 LR=0A00000005A6AE50 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.851Z] CR=4200004120040001 FPSCR=8202000000000000 XER=2004000182020000
[2023-12-15T15:16:12.851Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.851Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.851Z] FPR2 3fc7367ab58c488b (f: 3045869824.000000, d: 1.813501e-01)
[2023-12-15T15:16:12.851Z] FPR3 3f4a480000000000 (f: 0.000000, d: 8.020401e-04)
[2023-12-15T15:16:12.851Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.851Z] FPR5 40a6820000000000 (f: 0.000000, d: 2.881000e+03)
[2023-12-15T15:16:12.851Z] FPR6 4070b2623764c12d (f: 929349952.000000, d: 2.671490e+02)
[2023-12-15T15:16:12.851Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.851Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.851Z] FPR9 403187837ecdc890 (f: 2127415424.000000, d: 1.752935e+01)
[2023-12-15T15:16:12.851Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.851Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.851Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.851Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.851Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.851Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.851Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.851Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.851Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.851Z] ---------------------------------------
[2023-12-15T15:16:12.851Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.851Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.851Z] Unhandled exception
[2023-12-15T15:16:12.851Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.851Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.851Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.851Z] R0=0A0000000563D598 R1=0000010022E39660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.851Z] R4=09001000A1565D88 R5=0000010022ABAE00 R6=0000000000000003 R7=0A000000014C87B0
[2023-12-15T15:16:12.851Z] R8=0000010022ABAE00 R9=0000010021B98A00 R10=0000010022E398A0 R11=000001002C0DCB58
[2023-12-15T15:16:12.851Z] R12=0000000000000000 R13=0000010022E44800 R14=000001002C0DD420 R15=0000010022ABAE00
[2023-12-15T15:16:12.851Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=09001000A0E6D368
[2023-12-15T15:16:12.851Z] R20=0000000000000000 R21=0000000000000000 R22=0A000000014C86B0 R23=0A0000000563D548
[2023-12-15T15:16:12.851Z] R24=0A0000000563D580 R25=0000000000000001 R26=0000000000000000 R27=0000000000000000
[2023-12-15T15:16:12.851Z] R28=0A000000014C73E0 R29=0A0000000563D598 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.851Z] IAR=0A0000000563D598 LR=0A0000000563D598 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.851Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.851Z] Unhandled exception
[2023-12-15T15:16:12.851Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.851Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.851Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.851Z] R0=0A0000000194F628 R1=0000010022F99660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.851Z] R4=09001000A1565D88 R5=0000010022AAED00 R6=0000000000000003 R7=0A00000000586760
[2023-12-15T15:16:12.851Z] R8=0000010022AAED00 R9=0000010021B98A00 R10=0000010022F998A0 R11=00000100248737D0
[2023-12-15T15:16:12.851Z] R12=0000000000000000 R13=0000010022FA4800 R14=0000010024873B80 R15=0000010022AAED00
[2023-12-15T15:16:12.851Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=000000000000299E
[2023-12-15T15:16:12.851Z] R20=0A0000000025A850 R21=000000000000299E R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.851Z] R24=0A000000013B2ED8 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.851Z] R28=0A00000000000468 R29=0A00000000586660 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.851Z] IAR=0A0000000194F628 LR=0A0000000194F628 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.851Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.851Z] FPR0 000000000000005f (f: 95.000000, d: 4.693624e-322)
[2023-12-15T15:16:12.851Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.851Z] FPR2 3fc712c6df950b6e (f: 3751086848.000000, d: 1.802605e-01)
[2023-12-15T15:16:12.851Z] FPR3 3f82302000000000 (f: 0.000000, d: 8.880854e-03)
[2023-12-15T15:16:12.851Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.851Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.851Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.851Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.851Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.851Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.851Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.851Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.851Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.851Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.851Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR0 0000000000000060 (f: 96.000000, d: 4.743030e-322)
[2023-12-15T15:16:12.852Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.852Z] FPR2 3fc6e0cb1a6e1c46 (f: 443423808.000000, d: 1.787351e-01)
[2023-12-15T15:16:12.852Z] FPR3 3fa1d02000000000 (f: 0.000000, d: 3.479099e-02)
[2023-12-15T15:16:12.852Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.852Z] FPR5 40a6820000000000 (f: 0.000000, d: 2.881000e+03)
[2023-12-15T15:16:12.852Z] FPR6 4070b2623764c12d (f: 929349952.000000, d: 2.671490e+02)
[2023-12-15T15:16:12.852Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.852Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.852Z] FPR9 403187837ecdc890 (f: 2127415424.000000, d: 1.752935e+01)
[2023-12-15T15:16:12.852Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.852Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.852Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.852Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.852Z] Unhandled exception
[2023-12-15T15:16:12.852Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.852Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.852Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.852Z] R0=0A00000001911F28 R1=0000010022C29660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.852Z] R4=09001000A1565D88 R5=0000010022AB0300 R6=0000000000000003 R7=0A0000000058FAA8
[2023-12-15T15:16:12.852Z] R8=0000010022AB0300 R9=0000010021B98A00 R10=0000010022C298A0 R11=00000100248F99F0
[2023-12-15T15:16:12.852Z] R12=0000000000000000 R13=0000010022C34800 R14=00000100248F9DA0 R15=0000010022AB0300
[2023-12-15T15:16:12.852Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=0000000000002C7B
[2023-12-15T15:16:12.852Z] R20=0A0000000025A850 R21=0000000000002C7B R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.852Z] R24=0A0000000116C018 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.852Z] R28=0A00000001259FC0 R29=0A0000000025A850 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.852Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.852Z] IAR=0A00000001911F28 LR=0A00000001911F28 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.852Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.852Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.852Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.852Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.852Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.852Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.852Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.852Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.852Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.852Z] ---------------------------------------
[2023-12-15T15:16:12.852Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.852Z] Unhandled exception
[2023-12-15T15:16:12.852Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.852Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.852Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.852Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.852Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.852Z] R0=0000000000000001 R1=0000010023049660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.852Z] R4=0000010022B80400 R5=0000010010FD4C80 R6=0000010024741F80 R7=0000000000000088
[2023-12-15T15:16:12.852Z] R8=FFFFFFFFFFFFFFFF R9=00000100222ED238 R10=0A000200097166D5 R11=0000000000000001
[2023-12-15T15:16:12.852Z] R12=0000000042000041 R13=0000010023054800 R14=0000010024741FA0 R15=0000010022B80400
[2023-12-15T15:16:12.852Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=000000000000236C
[2023-12-15T15:16:12.852Z] R20=0A0000000025A850 R21=000000000000236C R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.852Z] R24=0A0000000130F3E0 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.852Z] R28=0A00000001259FC0 R29=0A0000000025A850 R30=0A00000001814DE8 R31=0000010021B6F900
[2023-12-15T15:16:12.852Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.852Z] IAR=0000000000000000 LR=0000000000000001 MSR=A00000000200D032 CTR=0000010010FD4C80
[2023-12-15T15:16:12.852Z] CR=4200004100000001 FPSCR=8202000000000000 XER=0000000182020000
[2023-12-15T15:16:12.852Z] FPR0 000000000000005f (f: 95.000000, d: 4.693624e-322)
[2023-12-15T15:16:12.852Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.852Z] FPR2 3fc730ecf4f9bc31 (f: 4110007296.000000, d: 1.811806e-01)
[2023-12-15T15:16:12.852Z] FPR3 3f58610000000000 (f: 0.000000, d: 1.487970e-03)
[2023-12-15T15:16:12.852Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.852Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.852Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.852Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.852Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.852Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.852Z] FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
[2023-12-15T15:16:12.852Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.853Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.853Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.853Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.853Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.853Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.853Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.853Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.853Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.853Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.853Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.853Z] ---------------------------------------
[2023-12-15T15:16:12.853Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.853Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.853Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.853Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.853Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.853Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.853Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.853Z] ---------------------------------------
[2023-12-15T15:16:12.853Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.853Z] Unhandled exception
[2023-12-15T15:16:12.853Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.853Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.853Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.853Z] FPR0 000000000000005f (f: 95.000000, d: 4.693624e-322)
[2023-12-15T15:16:12.853Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.853Z] FPR2 3fc7450658923208 (f: 1485976064.000000, d: 1.817940e-01)
[2023-12-15T15:16:12.853Z] FPR3 3ec2000000000000 (f: 0.000000, d: 2.145767e-06)
[2023-12-15T15:16:12.853Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.853Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.853Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.853Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.853Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.853Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.853Z] FPR10 407816625220ec6e (f: 1377889408.000000, d: 3.853990e+02)
[2023-12-15T15:16:12.853Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.853Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.853Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.853Z] R0=0A000000017F7048 R1=0000010023259660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.853Z] R4=09001000A1565D88 R5=0000010022338000 R6=0000000000000003 R7=0A0000000044D270
[2023-12-15T15:16:12.853Z] R8=0000010022338000 R9=0000010021B98A00 R10=00000100232598A0 R11=00000100246F4C10
[2023-12-15T15:16:12.853Z] R12=0000000000000000 R13=0000010023264800 R14=00000100246F4FC0 R15=0000010022338000
[2023-12-15T15:16:12.853Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=000000000000223D
[2023-12-15T15:16:12.853Z] R20=0A0000000025A850 R21=000000000000223D R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.853Z] R24=0A000000013CB670 R25=0000000000000000 R26=0000000000000001 R27=0000000000000001
[2023-12-15T15:16:12.853Z] R28=0A00000000000468 R29=0A0000000044D170 R30=00000100222A3200 R31=0000010021B6F900
[2023-12-15T15:16:12.853Z] IAR=0A000000017F7048 LR=0A000000017F7048 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.853Z] CR=42000241200ED660 FPSCR=8202000000000000 XER=200ED66082020000
[2023-12-15T15:16:12.853Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.853Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.853Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.853Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.853Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.853Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.853Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.853Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.853Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.853Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.853Z] ---------------------------------------
[2023-12-15T15:16:12.853Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.853Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.854Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.854Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.854Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.854Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.854Z] ---------------------------------------
[2023-12-15T15:16:12.854Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.854Z] FPR0 000000000000005f (f: 95.000000, d: 4.693624e-322)
[2023-12-15T15:16:12.854Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.854Z] FPR2 3fc73af9a6c5f71c (f: 2797991680.000000, d: 1.814873e-01)
[2023-12-15T15:16:12.854Z] FPR3 3f3a440000000000 (f: 0.000000, d: 4.007816e-04)
[2023-12-15T15:16:12.854Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.854Z] FPR5 40a6830000000000 (f: 0.000000, d: 2.881500e+03)
[2023-12-15T15:16:12.854Z] FPR6 4070b23633e586bc (f: 870680256.000000, d: 2.671382e+02)
[2023-12-15T15:16:12.854Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.854Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.854Z] FPR9 403179ac255aa460 (f: 626697344.000000, d: 1.747528e+01)
[2023-12-15T15:16:12.854Z] FPR10 c024000000000000 (f: 0.000000, d: -1.000000e+01)
[2023-12-15T15:16:12.854Z] FPR11 c30c6bf526340000 (f: 640942080.000000, d: -1.000000e+15)
[2023-12-15T15:16:12.854Z] FPR12 b9b4484bfeebc2a0 (f: 4276863744.000000, d: -1.000000e-30)
[2023-12-15T15:16:12.854Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.854Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.854Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.854Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.854Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.854Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.854Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.854Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.854Z] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch/core.20231215.101554.8323562.0003.dmp' in response to an event
[2023-12-15T15:16:12.854Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.854Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.854Z] ---------------------------------------
[2023-12-15T15:16:12.854Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:15:54 - please wait.
[2023-12-15T15:16:12.854Z] Unhandled exception
[2023-12-15T15:16:12.854Z] Type=Illegal instruction vmState=0x00000000
[2023-12-15T15:16:12.854Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
[2023-12-15T15:16:12.854Z] Handler1=09001000A1563880 Handler2=09001000A0EEA4C8
[2023-12-15T15:16:12.854Z] R0=0000010021B98A00 R1=00000100232FC660 R2=0000000000000001 R3=0000000000000001
[2023-12-15T15:16:12.854Z] R4=09001000A1565D88 R5=0000010022D87500 R6=0000000000000003 R7=0A00000000601E58
[2023-12-15T15:16:12.854Z] R8=0000010022D87500 R9=0000010021B98A00 R10=00000100232FC8A0 R11=0000010024BC6B70
[2023-12-15T15:16:12.854Z] R12=0000000000000000 R13=0000010023307800 R14=0000010024BC6F10 R15=0000010022D87500
[2023-12-15T15:16:12.854Z] R16=0000010020F7F478 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00000000FFFFFFFF
[2023-12-15T15:16:12.854Z] R20=0A0000000025A850 R21=0000000000007877 R22=0000000000000001 R23=0A00000000009FC8
[2023-12-15T15:16:12.854Z] R24=0900000005CF3F58 R25=0000000000000000 R26=0000000000000000 R27=0000000000000001
[2023-12-15T15:16:12.874Z] R28=0A00000000000468 R29=0A00000000601D58 R30=0000000000000000 R31=0000010021B6F900
[2023-12-15T15:16:12.874Z] IAR=0000010021B98A00 LR=0000010021B98A00 MSR=A00000000200D032 CTR=090000000A4C8180
[2023-12-15T15:16:12.874Z] CR=4200004120040000 FPSCR=8202000000000000 XER=2004000082020000
[2023-12-15T15:16:12.874Z] FPR0 000000000000005f (f: 95.000000, d: 4.693624e-322)
[2023-12-15T15:16:12.874Z] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2023-12-15T15:16:12.874Z] FPR2 3fc6d2832b3e6a3b (f: 725510720.000000, d: 1.782993e-01)
[2023-12-15T15:16:12.874Z] FPR3 3fa7348000000000 (f: 0.000000, d: 4.532242e-02)
[2023-12-15T15:16:12.874Z] FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
[2023-12-15T15:16:12.874Z] FPR5 40a6840000000000 (f: 0.000000, d: 2.882000e+03)
[2023-12-15T15:16:12.874Z] FPR6 4070b2158d12a02d (f: 2366808064.000000, d: 2.671303e+02)
[2023-12-15T15:16:12.874Z] FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
[2023-12-15T15:16:12.874Z] FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
[2023-12-15T15:16:12.874Z] FPR9 40316c8a96acbe50 (f: 2527903232.000000, d: 1.742399e+01)
[2023-12-15T15:16:12.874Z] FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
[2023-12-15T15:16:12.874Z] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2023-12-15T15:16:12.874Z] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2023-12-15T15:16:12.874Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-12-15T15:16:12.874Z] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch/core.20231215.101554.8323562.0003.dmp
[2023-12-15T15:16:12.874Z] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch/javacore.20231215.101554.8323562.0011.txt' in response to an event
[2023-12-15T15:16:12.874Z] Target=2_90_20231214_188 (AIX 7.2)
[2023-12-15T15:16:12.874Z] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
[2023-12-15T15:16:12.874Z] ----------- Stack Backtrace -----------
[2023-12-15T15:16:12.874Z] runJavaThread+0x28c (0x090000000A50E1F0 [libj9vm29.so+0x751f0])
[2023-12-15T15:16:12.874Z] _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000A49BFF4 [libj9vm29.so+0x2ff4])
[2023-12-15T15:16:12.874Z] omrsig_protect+0x4fc (0x0900000005EF41E0 [libj9prt29.so+0x5f1e0])
[2023-12-15T15:16:12.874Z] javaThreadProc+0x70 (0x090000000A49BE94 [libj9vm29.so+0x2e94])
[2023-12-15T15:16:12.874Z] thread_wrapper+0x14c (0x0900000004F20590 [libj9thr29.so+0x5590])
[2023-12-15T15:16:12.874Z] _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
[2023-12-15T15:16:12.874Z] ---------------------------------------
[2023-12-15T15:16:12.874Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/12/15 10:16:08 - please wait.
[2023-12-15T15:16:12.874Z] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch/core.20231215.101608.8323562.0012.dmp' in response to an event
[2023-12-15T15:16:12.875Z] 
[2023-12-15T15:16:12.875Z] 
[2023-12-15T15:16:12.875Z] *** Invalid JIT return address 0A00000001908988 in 0000010022EDCEF0
[2023-12-15T15:16:12.875Z] 
[2023-12-15T15:16:12.875Z] rerun:
[2023-12-15T15:16:12.875Z] cd /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/scratch && \
[2023-12-15T15:16:12.875Z] DISPLAY=unix:0 \
[2023-12-15T15:16:12.875Z] HOME=/home/jenkins \
[2023-12-15T15:16:12.875Z] LANG=en_US \
[2023-12-15T15:16:12.875Z] PATH=/bin:/usr/bin:/usr/sbin \
[2023-12-15T15:16:12.875Z] TZ=America/New_York \
[2023-12-15T15:16:12.875Z] CLASSPATH=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/jtreg.jar \
[2023-12-15T15:16:12.875Z]     /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java \
[2023-12-15T15:16:12.875Z]         -Dtest.vm.opts='-ea -esa -Xmx512m -Xverbosegclog -Xgcpolicy:nogc -Xmx2g' \
[2023-12-15T15:16:12.875Z]         -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-Xverbosegclog -J-Xgcpolicy:nogc -J-Xmx2g' \
[2023-12-15T15:16:12.875Z]         -Dtest.compiler.opts= \
[2023-12-15T15:16:12.875Z]         -Dtest.java.opts= \
[2023-12-15T15:16:12.875Z]         -Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
[2023-12-15T15:16:12.875Z]         -Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
[2023-12-15T15:16:12.875Z]         -Dtest.timeout.factor=8.0 \
[2023-12-15T15:16:12.875Z]         -Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \
[2023-12-15T15:16:12.875Z]         -Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \
[2023-12-15T15:16:12.875Z]         -Dtest.name=java/lang/Thread/virtual/stress/Skynet.java#default \
[2023-12-15T15:16:12.875Z]         -Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress/Skynet.java \
[2023-12-15T15:16:12.875Z]         -Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.875Z]         -Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.875Z]         -Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d \
[2023-12-15T15:16:12.875Z]         -Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d \
[2023-12-15T15:16:12.875Z]         -Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/classes/java/lang/Thread/virtual/stress/Skynet_default.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/Thread/virtual/stress \
[2023-12-15T15:16:12.875Z]         -ea \
[2023-12-15T15:16:12.875Z]         -esa \
[2023-12-15T15:16:12.875Z]         -Xmx512m \
[2023-12-15T15:16:12.875Z]         -Xverbosegclog \
[2023-12-15T15:16:12.875Z]         -Xgcpolicy:nogc \
[2023-12-15T15:16:12.875Z]         -Xmx2g \
[2023-12-15T15:16:12.875Z]         -Djava.library.path=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \
[2023-12-15T15:16:12.875Z]         -Xmx1g \
[2023-12-15T15:16:12.875Z]         -Xssi2K \
[2023-12-15T15:16:12.875Z]         -Xnocompressedrefs \
[2023-12-15T15:16:12.875Z]         com.sun.javatest.regtest.agent.MainWrapper /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work/java/lang/Thread/virtual/stress/Skynet_default.d/main.0.jta
[2023-12-15T15:16:12.875Z] 
[2023-12-15T15:16:12.875Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
[2023-12-15T15:16:12.875Z] --------------------------------------------------
[2023-12-15T15:16:12.875Z] Test results: failed: 1
[2023-12-15T15:16:36.781Z] Report written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/report/html/report.html
[2023-12-15T15:16:36.781Z] Results written to /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17026533377871/jdk_custom_0_ITER_1/work
[2023-12-15T15:16:36.781Z] Error: Some tests failed or other problems occurred.
[2023-12-15T15:16:36.781Z] 
[2023-12-15T15:16:36.781Z] jdk_custom_0_FAILED(ITER_1)
[2023-12-15T15:16:36.781Z] 
[2023-12-15T15:16:36.781Z] 
[2023-12-15T15:16:36.781Z] TEST TEARDOWN:
[2023-12-15T15:16:36.781Z] 
[2023-12-15T15:16:36.781Z] Attempting to destroy all caches in cacheDir /home/jenkins/.cache/javasharedresources/
[2023-12-15T15:16:36.781Z] 
[2023-12-15T15:16:36.781Z] JVMSHRC806I Compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
[2023-12-15T15:16:36.782Z] JVMSHRC807I Non-compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xcompressedrefs if you want to destroy a compressed references cache.
[2023-12-15T15:16:36.782Z] JVMSHRC005I No shared class caches available
[2023-12-15T15:16:36.782Z] cache cleanup done
babsingh commented 11 months ago

https://hyc-runtimes-jenkins.swg-devops.com/view/Test_grinder/job/Grinder/36805/consoleFull

In the grinder with -Xjit:verbose,vlog=vlog.txt,exclude={*} -Xnoaot, there are instances of the below message:

09:45:42  *** Invalid JIT return address 0000010023295420 in 0000010023182300

The failure happens while mounting a virtual thread. The virtual thread's state is stored in VirtualThread.cont.vmref, which is an instance of J9VMContinuation. During the mount phase, we swap the fields between J9VMThread and J9VMContinuation. !continuationstackslots <VirtualThread.cont.vmref> shows the stackslots of the unmounted virtual thread, which is to be mounted; in my local run with JIT enabled, this stack is heavily JIT optimized.

This failure probably happens because we fail to swap one or more J9VMThread field(s) (e.g. tempSlot, floatTemp*, jitReturnAddress) during the mount phase. I am unable to pin point these fields because dbx doesn't show me source code during the crash.

@hzongaro Are there any JIT + AIX specific fields in J9VMThread which need to be mirrored in J9VMContinuation?

hzongaro commented 11 months ago

Are there any JIT + AIX specific fields in J9VMThread which need to be mirrored in J9VMContinuation?

@zl-wang, are you aware of anything in this regard?

zl-wang commented 11 months ago

in my local run with JIT enabled, this stack is heavily JIT optimized.

Since there is no method JIT-ed (with that option shown), I don't see how java stack is JIT-optimized. @babsingh did you mean something else?

Are there any JIT + AIX specific fields in J9VMThread which need to be mirrored in J9VMContinuation?

JITToc might be AIX specific in J9VMThread (it is common with Linux BE). But it should have the same value as previous virtual thread (as long as it was set up correctly initially). I will quickly go through J9VMThread fields to see if there is any. Report back here later.

babsingh commented 11 months ago

Since there is no method JIT-ed (with that option shown), I don't see how java stack is JIT-optimized. @babsingh did you mean something else?

In my local run, the JIT was fully enabled. Locally, I couldn't reproduce the error with -Xjit:verbose,vlog=vlog.txt,exclude={*} -Xnoaot. I mentioned that grinder w.r.t. Invalid JIT return address; are there other reasons for Invalid JIT return address if there is no method JIT'ed?

zl-wang commented 11 months ago

as far as I can see, J9VMThread AIX specific fields:

        UDATA jitTOC;
        UDATA jitCurrentRIFlags;
        UDATA jitPendingRIFlags;
        struct J9RIParameters *riParameters;

all these fields are only relevant, when there are JIT-ed code running. (there is codertTOC field which seemed used by Z system only). And, I knew the latter 3 fields are not used at all nowadays (RI was disabled 3 years ago).

zl-wang commented 11 months ago

are there other reasons for Invalid JIT return address if there is no method JIT'ed?

No, I cannot think of one. Could you look into where Invalid JIT return address can be emitted? maybe that can lead to something. If I remembered correctly, the check happened only at the point Interpreter is about to return to JIT-ed code (i.e. caller). How this can happen without JIT-ed code at all might be the thing ultimately.

pshipton commented 11 months ago

What about if something is corrupted and the VM tries to return to JIT-ed code that doesn't actually exist.

gita-omr commented 11 months ago

I think it would be good to enable some stack walking tracing. For example, I can see something like:

Trc_VRB_WalkStackFrames_Entry inside walkStackFrames().

But I am not sure how to enable it. Maybe some other VM tracing during stack unwinding would be useful?

pshipton commented 10 months ago

Stack walk tracing is enabled via -verbose:stackwalk.

gita-omr commented 10 months ago

I've been running grinder [1] with -Xjit:verbose,exclude={*} -Xnoaot -verbose:stackwalk with the build [2] that has a fix for [3] and I don't see any failures so far.

[1] https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/37043/ [2] https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK21_ppc64_aix_Personal/92/OpenJ9-JDK21-ppc64_aix-20231218-173701.tar.gz [3] https://github.com/eclipse-openj9/openj9/pull/18637

gita-omr commented 10 months ago

Correction to the above: the tests did fail but due to a timeout. Running without -verbose:stackwalknow.

babsingh commented 10 months ago

[ISSUE 2] ... crash in runJavaThread

I was able to get a better native stack for the runJavaThread crash. After looking at the core file in jdmpview and dbx, it was found that the current approach of recycling continuations retains old values which leads to segfaults when a stack grow is triggered by a JIT StackOverflow. Couldn't figure out why the runJavaThread crash is only seen on AIX, but all issues related to the crash are resolved with the fix proposed in https://github.com/eclipse-openj9/openj9/pull/18649. Successfully ran Skynet 50 times locally on AIX and verified that #18649 resolves the runJavaThread crash.

...
generateDiagnosticFiles(??, ??), line 1162 in "gphandle.c"
omrsig_protect(??, ??, ??, ??, ??, ??, ??), line 425 in "omrsignal.c"
structuredSignalHandler(??, ??, ??, ??), line 837 in "gphandle.c"
mainSynchSignalHandler(??, ??, ??), line 1066 in "omrsignal.c"
growSlotIterator(J9VMThread*, J9StackWalkState*, J9Object**, void const*)(0x10035982000, 0x10037da9230, 0xb7654321bd90aba4, 0xb7654321bd90aba4), line 524 in "growstack.cpp"
IPRA.$walkMethodFrame(??), line 1585 in "swalk.c"
walkStackFrames(??, ??), line 331 in "swalk.c"
internalGrowJavaStack(J9VMThread*, unsigned long)(??, ??), line 235 in "growstack.cpp"
growJavaStack(??, ??), line 97 in "growstack.cpp"
old_slow_jitStackOverflow(??), line 1347 in "cnathelp.cpp"
jitStackOverflow(), line 1956 in "pnathelp.s"
runJavaThread(??), line 682 in "callin.cpp"
javaProtectedThreadProc(J9PortLibrary*, void*)(??, ??), line 2104 in "vmthread.cpp"
omrsig_protect(??, ??, ??, ??, ??, ??, ??), line 425 in "omrsignal.c"
javaThreadProc(??), line 383 in "vmthread.cpp"
thread_wrapper(??), line 1733 in "omrthread.c"
gita-omr commented 10 months ago

Excellent! And just to confirm: with -Xjit:verbose,exclude={*} -Xnoaot the test case timeouts sometimes but generally passes.

babsingh commented 10 months ago

re https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1843799810:

Fix for ISSUE 1 and ISSUE 3

@LinHu2016 Can you confirm if issues 1 and 2 in https://github.com/eclipse-openj9/openj9/issues/16728#issuecomment-1843799810 are fixed by the above PRs?

This issue can be closed after the below PRs are merged.

Fix for ISSUE 2

Enable Skynet on AIX

LinHu2016 commented 10 months ago

https://github.com/eclipse-openj9/openj9/pull/18637 https://github.com/eclipse-openj9/openj9/pull/18651 https://github.com/eclipse-openj9/openj9/pull/18652

@babsingh yes, ISSUE 1 and ISSUE 3 are fixed by the above PRs.