search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JDK 19 MiniMix_5m_1 hang in Thread.getStateImpl #16036

Closed pshipton closed 2 years ago

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19 MiniMix_5m_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19/system_test_output.tar.gz

LT  04:08:01.390 - Completed 13.4%. Number of tests started=10586 (+3937)
STF 04:12:19.367 - Heartbeat: Process LT  is still running
STF 04:17:19.211 - Heartbeat: Process LT  is still running
STF 04:22:19.069 - Heartbeat: Process LT  is still running
STF 04:27:19.047 - Heartbeat: Process LT  is still running
STF 04:32:19.283 - Heartbeat: Process LT  is still running
STF 04:37:19.023 - Heartbeat: Process LT  is still running
STF 04:42:19.203 - Heartbeat: Process LT  is still running
STF 04:47:19.411 - Heartbeat: Process LT  is still running
STF 04:52:19.009 - Heartbeat: Process LT  is still running
STF 04:57:19.398 - Heartbeat: Process LT  is still running
STF 05:02:19.487 - Heartbeat: Process LT  is still running
STF 05:07:19.427 - Heartbeat: Process LT  is still running
STF 05:12:19.456 - Heartbeat: Process LT  is still running
STF 05:12:20.457 - **FAILED** Process LT  has timed out
3XMTHREADINFO      "load-9" J9VMThread:0x000000000060BE00, omrthread_t:0x000003FF8C5B5040, java/lang/Thread:0x000000003A012EF0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x21, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x0000000039A7D7F8)
3XMTHREADINFO1            (native thread ID:0xEE4E, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00001001)
3XMTHREADINFO2            (native stack address range from:0x000003FF6C1BC000, to:0x000003FF6C1FC000, size:0x40000)
3XMCPUTIME               CPU usage total: 3.111664071 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=2331328 (0x2392C0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.getStateImpl(Native Method)
4XESTACKTRACE                at java/lang/Thread.threadState(Thread.java:2864(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x000000003A07F8C8, entry count: 1)
4XESTACKTRACE                at java/lang/Thread.isTerminated(Thread.java:2872(Compiled Code))
4XESTACKTRACE                at java/lang/Thread.getThreadGroup(Thread.java:2025(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadGroup.enumerate(ThreadGroup.java:450)
4XESTACKTRACE                at gnu/testlet/java/lang/ThreadGroup/enumerate.test(enumerate.java:48)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$DMH/0x00000000d80054f0.invokeVirtual(LambdaForm$DMH(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x00000000d8005be0.invoke(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x000000008c40ee70.invokeExact_MT(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155(Compiled Code))
4XESTACKTRACE                at jdk/internal/reflect/DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:578(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/loadTest/adaptors/MauveAdaptor.executeTest(MauveAdaptor.java:74(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/loadTest/LoadTestRunner$2.run(LoadTestRunner.java:182)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:1573)
pshipton commented 2 years ago

@babsingh @tajila fyi

tajila commented 2 years ago

@ChengJin01 Please take a look at this

ChengJin01 commented 2 years ago

1) I went through the dumps at https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/19/system_test_output.tar.gz and didn't find any suspicious thread that might trigger the hang in the failing job. In addition, there was no deadlock detected via jdmpview.

As for the J9VMThread load-9 mentioned in the description, it was waiting for an unowned mutex lock, which doesn't seem to be hung in there.

2LKREGMON          Thread public flags mutex lock (0x000003FF8C421268): <unowned> <------
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "load-9" (J9VMThread:0x000000000060BE00)

2) I manually ran the failing test suite MiniMix_5m_1 with the failing build over 10 times on one of Jenkins zLinux machines and it always passed without any issue. 3) There is no hang detected so far in the Grinder (X50/still ongoing) at https://openj9-jenkins.osuosl.org/job/Grinder/1321/consoleText. Will wait and see from there how frequently the hang occurs.

pshipton commented 2 years ago

There are a couple of javacore files, taken a minute apart, and both showed the same stack for the "load-9" thread.

ChengJin01 commented 2 years ago

Assuming there was something happening for "load-9" thread according to the javacores,

3XMTHREADINFO      "load-9" J9VMThread:0x000000000060BE00, omrthread_t:0x000003FF8C5B5040, java/lang/Thread:0x000000003A012EF0, state:R, prio=5
...
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.getStateImpl(Native Method) <------
4XESTACKTRACE                at java/lang/Thread.threadState(Thread.java:2864(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x000000003A07F8C8, entry count: 1)
4XESTACKTRACE                at java/lang/Thread.isTerminated(Thread.java:2872(Compiled Code))
4XESTACKTRACE                at java/lang/Thread.getThreadGroup(Thread.java:2025(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadGroup.enumerate(ThreadGroup.java:450)
4XESTACKTRACE                at gnu/testlet/java/lang/ThreadGroup/enumerate.test(enumerate.java:48)
...
3XMTHREADINFO3           Native callstack:
...
4XENATIVESTACK                (0x000003FF6C1F9B10 [<unknown>+0x0])
4XENATIVESTACK               pthread_cond_wait+0x138 (0x000003FF9108C3B0 [libpthread.so.0+0xc3b0])
4XENATIVESTACK                (0x000003FF90607B26 [libj9thr29.so+0x7b26])
4XENATIVESTACK               omrthread_monitor_wait+0x60 (0x000003FF9060B690 [libj9thr29.so+0xb690])
4XENATIVESTACK                (0x000003FF90818D9A [libj9vm29.so+0x98d9a]) <--- internalAcquireVMAccessNoMutexWithMask
4XENATIVESTACK                (0x000003FF9081C8EA [libj9vm29.so+0x9c8ea]) <--- haltThreadForInspection (via objdump)
4XENATIVESTACK               Java_java_lang_Thread_getStateImpl+0x90 (0x000003FF8AC55E08 [libjclse29.so+0x55e08])
4XENATIVESTACK                (0x000003FF6FB5839A [<unknown>+0x0])

it should be stuck at the native method getStateImpl() in Thread.threadState at https://github.com/ibmruntimes/openj9-openjdk-jdk19/blob/b9e3daf582ff40b3a187281241c3cb691e4fec73/src/java.base/share/classes/java/lang/Thread.java#L2864

    boolean isTerminated() {
        return threadState() == State.TERMINATED;
    }

    State threadState() {
        synchronized (interruptLock) {
            if (eetop == NO_REF) {
                if (isDead()) {
                    return State.TERMINATED;
                }
                return State.NEW;
            }
            return State.values()[getStateImpl(eetop)]; <-------
        }
    }

where the native code of getStateImpl was at https://github.com/eclipse-openj9/openj9/blob/96fe9b858827d90eb7ad49817a29b0da2e73c2b5/runtime/jcl/common/thread.cpp#L87

Java_java_lang_Thread_getStateImpl(JNIEnv *env, jobject recv, jlong threadRef)
{
...
    Trc_JCL_Thread_getStateImpl_Entry(currentThread, vmThread);
    currentThread->javaVM->internalVMFunctions->internalEnterVMFromJNI(currentThread);
    currentThread->javaVM->internalVMFunctions->haltThreadForInspection(currentThread, vmThread); <------

and https://github.com/eclipse-openj9/openj9/blob/e6744487bdab6ba75dd7299ba98fff260bfbba2e/runtime/vm/VMAccess.cpp#L1296

haltThreadForInspection(J9VMThread * currentThread, J9VMThread * vmThread)
{

_tryAgain:

    Assert_VM_mustHaveVMAccess(currentThread);

    /* Inspecting the current thread does not require any halting */
    if (currentThread != vmThread) {
        VM_VMAccess::setPublicFlags(currentThread, J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT);

        omrthread_monitor_enter(vmThread->publicFlagsMutex);

        /* increment the inspection count but don't try to short circuit -- the thread might not actually be halted yet */
        vmThread->inspectionSuspendCount += 1;

        /* Now halt the thread for inspection */
        setHaltFlag(vmThread, J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION);

        /* If the thread doesn't have VM access and it not queued for exclusive we can proceed immediately */
        if (vmThread->publicFlags & (J9_PUBLIC_FLAGS_VM_ACCESS | J9_PUBLIC_FLAGS_QUEUED_FOR_EXCLUSIVE)) {
            /* Release VM access while waiting */
            /* (We must release the other thread's publicFlagsMutex to avoid deadlock here) */
            omrthread_monitor_exit(vmThread->publicFlagsMutex);
            internalReleaseVMAccess(currentThread);
            omrthread_monitor_enter(vmThread->publicFlagsMutex);
#if defined(J9VM_INTERP_ATOMIC_FREE_JNI)
#if defined(J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH)
            flushProcessWriteBuffers(currentThread->javaVM);
#endif /* J9VM_INTERP_ATOMIC_FREE_JNI_USES_FLUSH */
            VM_AtomicSupport::readWriteBarrier(); // necessary?
#endif /* J9VM_INTERP_ATOMIC_FREE_JNI */
            if (VM_VMAccess::mustWaitForVMAccessRelease(vmThread)) {
                while (J9_ARE_ANY_BITS_SET(vmThread->publicFlags, J9_PUBLIC_FLAGS_VM_ACCESS | J9_PUBLIC_FLAGS_QUEUED_FOR_EXCLUSIVE)) {
                    omrthread_monitor_wait(vmThread->publicFlagsMutex);
                }
            }
            omrthread_monitor_exit(vmThread->publicFlagsMutex);

            /* Thread is halted - reacquire VM access */

            omrthread_monitor_enter(currentThread->publicFlagsMutex);
----------->     internalAcquireVMAccessNoMutexWithMask(currentThread, J9_PUBLIC_FLAGS_HALT_THREAD_ANY - J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION);
            omrthread_monitor_exit(currentThread->publicFlagsMutex);
ChengJin01 commented 2 years ago

It seems haltThreadForInspection was only modified via a PR at https://github.com/eclipse-openj9/openj9/pull/15227 recently to resolve a couple of issues related to hang/halted thread. So I am wondering whether there was any connection between this issue and https://github.com/eclipse-openj9/openj9/pull/15227.

ChengJin01 commented 2 years ago

FYI: @amicic, @gacholio

ChengJin01 commented 2 years ago

Launched another Grinder (x200) at https://openj9-jenkins.osuosl.org/job/Grinder/1327/ with the latest passing build at https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/167/ to see how it goes.

tajila commented 2 years ago

@ChengJin01 Do you know if this fails on JDK17?

ChengJin01 commented 2 years ago

@ChengJin01 Do you know if this fails on JDK17?

This job never fails on JDK17 in recent histories at https://openj9-jenkins.osuosl.org/view/Test_System_Nightly/job/Test_openjdk17_j9_extended.system_s390x_linux_Nightly/, but we can launch a Grinder with the latest build to double-check.

ChengJin01 commented 2 years ago

Launched a Grinder (x200) at https://openj9-jenkins.osuosl.org/job/Grinder/1329/ with the latest build at https://openj9-jenkins.osuosl.org/view/Test_System_Nightly/job/Test_openjdk17_j9_extended.system_s390x_linux_Nightly/328/.

ChengJin01 commented 2 years ago

The previously completed Grinder at https://openj9-jenkins.osuosl.org/job/Grinder/1327/ ended without any issue (killed by after 98 runs due to timeout) at https://openj9-jenkins.osuosl.org/job/Grinder_testList_0/161/consoleText

[2022-10-07T02:18:56.782Z] LT  02:18:54.278 - Completed 46.9%. Number of tests started=34837 (+5415)
[2022-10-07T02:19:15.668Z] Cancelling nested steps due to timeout
[2022-10-07T02:19:15.835Z] Sending interrupt signal to process
[2022-10-07T02:19:17.071Z] LT  02:19:14.265 - Completed 53.6%. Number of tests started=39086 (+4249)

So it seems the hang was only triggered by the changes between https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/168/

OpenJ9: [c19e4267798](https://github.com/eclipse-openj9/openj9/commit/c19e4267798b13e6ee21c71290a2877c07fc4abb)
OMR: [b8f2f89464c](https://github.com/eclipse-openj9/openj9-omr/commit/b8f2f89464ca71f38dd7d63f49bd3fc0f945224d)
OpenJDK19: [b9e3daf582f](https://github.com/ibmruntimes/openj9-openjdk-jdk19/commit/b9e3daf582ff40b3a187281241c3cb691e4fec73) - openj9-openjdk-jdk19

and https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/167/

OpenJ9: [710fd952924](https://github.com/eclipse-openj9/openj9/commit/710fd952924c673ee702fcbc311c99d70095ecde)
OMR: [f99c9dd2248](https://github.com/eclipse-openj9/openj9-omr/commit/f99c9dd2248e6ee35e40536b3c243eee33924407)
OpenJDK19: [05db3773ec3](https://github.com/ibmruntimes/openj9-openjdk-jdk19/commit/05db3773ec323ac48a721cc22e59b0a768e5d4e7) - openj9-openjdk-jdk19
gacholio commented 2 years ago

If I understand the above comments, this rules out https://github.com/eclipse-openj9/openj9/pull/15227 as the cause (committed in June).

It would be helpful to see the !threads stack and !threads flags output.

ChengJin01 commented 2 years ago

There was no hang detected in the Grinder at https://openj9-jenkins.osuosl.org/job/Grinder/1329/ for JDK17. So the problem is most likely related to Project Loom in JDK19 for the merged changes in OpenJ9 between https://github.com/eclipse-openj9/openj9/commit/c19e4267798b13e6ee21c71290a2877c07fc4abb and https://github.com/eclipse-openj9/openj9/commit/710fd952924c673ee702fcbc311c99d70095ecde given there is nothing suspicious in OMR changes.

FYI: @fengxue-IS

gacholio commented 2 years ago

https://github.com/eclipse-openj9/openj9/compare/710fd952924c673ee702fcbc311c99d70095ecde...c19e4267798b13e6ee21c71290a2877c07fc4abb

gacholio commented 2 years ago

Nothing immediately suspicious - does the test fail with -Xint or on platforms other than 390?

ChengJin01 commented 2 years ago

Just launched a couple of Grinders (x100) with the latest build on other platforms plus the failing build with -Xint on Linux/s390x see how it goes. https://openj9-jenkins.osuosl.org/job/Grinder/1349/ (Linux/x86_64) https://openj9-jenkins.osuosl.org/job/Grinder/1350/ (macOS/Aarch64) https://openj9-jenkins.osuosl.org/job/Grinder/1351/ (AIX/PPC64) https://openj9-jenkins.osuosl.org/job/Grinder/1352/ (with -Xint on Linux/s390x)

pshipton commented 2 years ago

So far we've only ever seen this hang once, in jdk19 nightly builds.

ChengJin01 commented 2 years ago

All grinders at https://github.com/eclipse-openj9/openj9/issues/16036#issuecomment-1274917568 passed without any issue: https://openj9-jenkins.osuosl.org/job/Grinder/1352/tapTestReport/ (with -Xint on Linux/s390x) https://openj9-jenkins.osuosl.org/job/Grinder/1351/tapTestReport/ (AIX/PPC64) https://openj9-jenkins.osuosl.org/job/Grinder/1350/tapTestReport/ (macOS/Aarch64) https://openj9-jenkins.osuosl.org/job/Grinder/1349/tapTestReport/ (Linux/x86_64)

So it seems the hang only occurred on Linux/s390x with JIT enabled and was captured only once in Grinder(x50) at https://openj9-jenkins.osuosl.org/job/Grinder/1321/consoleText.

gacholio commented 2 years ago

If that's the case, getting the information I requested earlier (thread flags, java stacks and native stacks) will help us figure out what's going on.

ChengJin01 commented 2 years ago

1) The flags of threads in the core don't seem to be problematic as follows (in which the majority of threads parked in there)

> !threads flags
Attached Threads List. For more options, run !threads help

    !j9vmthread 0x69700 publicFlags=1 privateFlags=1008 inNative=0 // main <---J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x6bd00 publicFlags=1 privateFlags=101a inNative=0 // JIT Compilation Thread-000
    !j9vmthread 0x6e300 publicFlags=1 privateFlags=101a inNative=0 // JIT Compilation Thread-001
    !j9vmthread 0x204900 publicFlags=49000 privateFlags=0 inNative=0 // load-0 <-- J9_PUBLIC_FLAGS_EXCLUSIVE_SET_NOT_SAFE  |  J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION  |  J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT 
    !j9vmthread 0x5f9000 publicFlags=1 privateFlags=0 inNative=0 // load-1
    !j9vmthread 0x5fb600 publicFlags=1 privateFlags=0 inNative=0 // load-2
    !j9vmthread 0x5fdb00 publicFlags=20001 privateFlags=0 inNative=0 // load-3 <--- J9_PUBLIC_FLAGS_THREAD_PARKED  | J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x600100 publicFlags=20001 privateFlags=0 inNative=0 // load-4
    !j9vmthread 0x602700 publicFlags=1 privateFlags=0 inNative=0 // load-5
    !j9vmthread 0x604d00 publicFlags=20001 privateFlags=0 inNative=0 // load-6
    !j9vmthread 0x607300 publicFlags=20001 privateFlags=0 inNative=0 // load-7
    !j9vmthread 0x609900 publicFlags=20001 privateFlags=0 inNative=0 // load-8
----> !j9vmthread 0x60be00 publicFlags=1001 privateFlags=0 inNative=0 // load-9 <---- J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT |  J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 
    !j9vmthread 0x6ef800 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-1
    !j9vmthread 0x6fe300 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-2
    !j9vmthread 0x737500 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-3
    !j9vmthread 0x745c00 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-4
    !j9vmthread 0x748200 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-5
    !j9vmthread 0x74e800 publicFlags=20001 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-6
    !j9vmthread 0x754e00 publicFlags=1 privateFlags=2 inNative=0 // ForkJoinPool-2-worker-8
...

define J9_PUBLIC_FLAGS_HALT_THREAD_EXCLUSIVE 0x1

define J9_PUBLIC_FLAGS_NOT_AT_SAFE_POINT 0x1000

define J9_PUBLIC_FLAGS_THREAD_PARKED 0x20000

define J9_PUBLIC_FLAGS_EXCLUSIVE_SET_NOT_SAFE 0x40000

define J9_PUBLIC_FLAGS_HALT_THREAD_INSPECTION 0x8000

But the java stacktrace of thread load-9 mentioned in the description indicates it was totally in JIT:

> !stackslots 0x60be00
<60be00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x000000000060BE00 ***
<60be00>        ITERATE_O_SLOTS
<60be00>        RECORD_BYTECODE_PC_OFFSET
<60be00> Initial values: walkSP = 0x00000000006227E0, PC = 0x0000000000000006, literals = 0x0000000000000000, A0 = 0x0000000000622948, j2iFrame = 0x00000000006229C8, ELS = 0x000003FF6C1FAA30, decomp = 0x0000000000000000
<60be00> JIT JNI call-out frame: bp = 0x0000000000622800, sp = 0x00000000006227E0, pc = 0x0000000000000006, cp = 0x00000000000B8C90, arg0EA = 0x0000000000622948, flags = 0x0000000020000000
<60be00>        Method: java/lang/Thread.getStateImpl(J)I !j9method 0x00000000000BB130
<60be00> JIT inline frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.threadState()Ljava/lang/Thread$State; !j9method 0x00000000000BAD90
<60be00>        Bytecode index = 43, inlineDepth = 2, PC offset = 0x000003FF6FB5764D
<60be00> JIT inline frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.isTerminated()Z !j9method 0x00000000000BADB0
<60be00>        Bytecode index = 1, inlineDepth = 1, PC offset = 0xFFFFFFFFFFFFF2A0
<60be00> JIT frame: bp = 0x00000000006228D8, pc = 0x000003FF6FB5839A, unwindSP = 0x0000000000622808, cp = 0x00000000000B8C90, arg0EA = 0x00000000006228E0, jitInfo = 0x000003FF6C260738
<60be00>        Method: java/lang/Thread.getThreadGroup()Ljava/lang/ThreadGroup; !j9method 0x00000000000BA990
<60be00>        Bytecode index = 1, inlineDepth = 0, PC offset = 0x00000000000001A0
<60be00>        stackMap=0x000003FF6C260B78, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFB8)
<60be00>        Described JIT args starting at 0x00000000006228E0 for U16(0x0001) slots
<60be00>                O-Slot: : a0[0x00000000006228E0] = 0x000000003A012430
<60be00>        Described JIT temps starting at 0x0000000000622890 for IDATA(0x0000000000000009) slots
<60be00>                O-Slot: : t8[0x0000000000622890] = 0x000000003A07F8C8
<60be00>                O-Slot: : t7[0x0000000000622898] = 0x000000003A07F8C8
<60be00>                I-Slot: : t6[0x00000000006228A0] = 0x00000000000BACD0
<60be00>                O-Slot: : t5[0x00000000006228A8] = 0x000000003A012430
<60be00>                I-Slot: : t4[0x00000000006228B0] = 0x00000000006228E8
<60be00>                O-Slot: : t3[0x00000000006228B8] = 0x000000003A012430
<60be00>                O-Slot: : t2[0x00000000006228C0] = 0x000000003A07F8C8
<60be00>                O-Slot: : t1[0x00000000006228C8] = 0x00000000B0472C00
<60be00>                O-Slot: : t0[0x00000000006228D0] = 0x000000003A012430
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622838] = UDATA(0x000003FF6F644F70) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622840] = UDATA(0x0000000000E16900) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622848] = UDATA(0x00000000B0472108) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622850] = UDATA(0x0000000000342C00) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622858] = UDATA(0x0000000000000000) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622860] = UDATA(0x00000000B0472108) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622868] = UDATA(0x00000000B0472050) (jit_r12)
<60be00> I2J values: PC = 0x000003FF354F5CA5, A0 = 0x0000000000622948, walkSP = 0x00000000006228F0, literals = 0x00000000000DC7B8, JIT PC = 0x000003FF8BCDD7C0, pcAddress = 0x000003FF6C1FAA58, decomp = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622900, sp = 0x00000000006228F0, pc = 0x000003FF354F5CA5, cp = 0x00000000000DBD90, arg0EA = 0x0000000000622948, flags = 0x0000000000000000
<60be00>        Method: java/lang/ThreadGroup.enumerate([Ljava/lang/Thread;Z)I !j9method 0x00000000000DC7B8
<60be00>        Bytecode index = 45
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622948 for 0x0000000000000009 slots
<60be00>                O-Slot: a0[0x0000000000622948] = 0x00000000B046DC10
<60be00>                O-Slot: a1[0x0000000000622940] = 0x00000000B0471C18
<60be00>                I-Slot: a2[0x0000000000622938] = 0x00000001B046F558
<60be00>                I-Slot: t3[0x0000000000622930] = 0x0000000000000000
<60be00>                O-Slot: t4[0x0000000000622928] = 0x00000000B0472598
<60be00>                I-Slot: t5[0x0000000000622920] = 0x000000E1B0472598
<60be00>                I-Slot: t6[0x0000000000622918] = 0x0000001200000000
<60be00>                O-Slot: t7[0x0000000000622910] = 0x000000003A012430
<60be00>                I-Slot: t8[0x0000000000622908] = 0x0000000039A7EFB0
<60be00> J2I frame: bp = 0x00000000006229C8, sp = 0x0000000000622950, pc = 0x000003FF0011E54F, cp = 0x0000000000E61870, arg0EA = 0x0000000000622A08, flags = 0x0000000010000000
<60be00>        Method: gnu/testlet/java/lang/ThreadGroup/enumerate.test(Lgnu/testlet/TestHarness;)V !j9method 0x0000000000E612C8
<60be00>        Bytecode index = 75
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622A08 for 0x0000000000000008 slots
<60be00>                I-Slot: a0[0x0000000000622A08] = 0x00000000B046D0B0
<60be00>                O-Slot: a1[0x0000000000622A00] = 0x00000000B046DAA0
<60be00>                I-Slot: t2[0x00000000006229F8] = 0x0000000039A7EFB0
<60be00>                I-Slot: t3[0x00000000006229F0] = 0x00000000B046DC10
<60be00>                I-Slot: t4[0x00000000006229E8] = 0x00000000B046DC50
<60be00>                I-Slot: t5[0x00000000006229E0] = 0x00000000B046E460
<60be00>                I-Slot: t6[0x00000000006229D8] = 0x0000000A00000000
<60be00>                I-Slot: t7[0x00000000006229D0] = 0x00000000B0471C18
<60be00>        JIT-J2I-RegisterMap[0x00000000006229A8] = UDATA(0x000000003C4EC4B8) (jit_r6)
<60be00>        JIT-J2I-RegisterMap[0x00000000006229A0] = UDATA(0x000003FF6F7482F4) (jit_r7)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622998] = UDATA(0x000000000008E200) (jit_r8)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622990] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622988] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-J2I-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT frame: bp = 0x0000000000622A80, pc = 0x000003FF6FA4876A, unwindSP = 0x0000000000622A00, cp = 0x00000000005FF4C0, arg0EA = 0x0000000000622A98, jitInfo = 0x000003FF6D0C99F8
<60be00>        Method: java/lang/invoke/LambdaForm$DMH/0x00000000d80054f0.invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V !j9method 0x00000000005FF5C8
<60be00>        Bytecode index = 11, inlineDepth = 0, PC offset = 0x000000000000017C
<60be00>        stackMap=0x000003FF6D0C9B20, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFF8)
<60be00>        Described JIT args starting at 0x0000000000622A88 for U16(0x0003) slots
<60be00>                O-Slot: : a2[0x0000000000622A88] = 0x00000000B046DAA0
<60be00>                O-Slot: : a1[0x0000000000622A90] = 0x00000000B046D0B0
<60be00>                O-Slot: : a0[0x0000000000622A98] = 0x000000003C4EC548
<60be00>        Described JIT temps starting at 0x0000000000622A78 for IDATA(0x0000000000000001) slots
<60be00>                O-Slot: : t0[0x0000000000622A78] = 0x000000003C4EC5D8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A20] = UDATA(0x000000003C4EC4B8) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A28] = UDATA(0x000003FF6F7482F4) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A30] = UDATA(0x000000000008E200) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622A38] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622988] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT frame: bp = 0x0000000000622B78, pc = 0x000003FF6F9A761C, unwindSP = 0x0000000000622A88, cp = 0x00000000006010C0, arg0EA = 0x0000000000622B90, jitInfo = 0x000003FF6D0D78B8
<60be00>        Method: java/lang/invoke/LambdaForm$MH/0x00000000d8005be0.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000006011E8
<60be00>        Bytecode index = 52, inlineDepth = 0, PC offset = 0x000000000000022E
<60be00>        stackMap=0x000003FF6D0D7A30, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFF78)
<60be00>        Described JIT args starting at 0x0000000000622B80 for U16(0x0003) slots
<60be00>                O-Slot: : a2[0x0000000000622B80] = 0x00000000B046DAA0
<60be00>                O-Slot: : a1[0x0000000000622B88] = 0x00000000B046D0B0
<60be00>                O-Slot: : a0[0x0000000000622B90] = 0x000000003C4EC4B8
<60be00>        Described JIT temps starting at 0x0000000000622AF0 for IDATA(0x0000000000000011) slots
<60be00>                O-Slot: : t16[0x0000000000622AF0] = 0x00000000B046DAA0
<60be00>                O-Slot: : t15[0x0000000000622AF8] = 0x00000000B046D0B0
<60be00>                O-Slot: : t14[0x0000000000622B00] = 0x000000003C4EC548
<60be00>                O-Slot: : t13[0x0000000000622B08] = 0x00000000B046DAA0
<60be00>                O-Slot: : t12[0x0000000000622B10] = 0x00000000B046D0B0
<60be00>                O-Slot: : t11[0x0000000000622B18] = 0x000000003C4EC548
<60be00>                O-Slot: : t10[0x0000000000622B20] = 0x00000000B046DAA0
<60be00>                O-Slot: : t9[0x0000000000622B28] = 0x00000000B046DAA0
<60be00>                O-Slot: : t8[0x0000000000622B30] = 0x000000003C4EC578
<60be00>                O-Slot: : t7[0x0000000000622B38] = 0x00000000B046D0B0
<60be00>                O-Slot: : t6[0x0000000000622B40] = 0x00000000B046D0B0
<60be00>                O-Slot: : t5[0x0000000000622B48] = 0x000000003C4EC5A8
<60be00>                O-Slot: : t4[0x0000000000622B50] = 0x000000003C4EC548
<60be00>                O-Slot: : t3[0x0000000000622B58] = 0x00000000B046DAA0
<60be00>                O-Slot: : t2[0x0000000000622B60] = 0x000000003C4EC578
<60be00>                O-Slot: : t1[0x0000000000622B68] = 0x00000000B046D0B0
<60be00>                O-Slot: : t0[0x0000000000622B70] = 0x000000003C4EC5A8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000000000)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AA0] = UDATA(0x000003FF6F9A7400) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AA8] = UDATA(0x00000000B046DAA0) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AB0] = UDATA(0x00000000B046D0B0) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AB8] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622AC0] = UDATA(0x00000000B046DAA0) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622980] = UDATA(0x000000003C4EC4B8) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622978] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000004107C0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: java/lang/invoke/LambdaForm$MH/0x000000008c40ee70.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000004108D8
<60be00>        Bytecode index = 20, inlineDepth = 4, PC offset = 0x000003FF6FA8C321
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000001F9EC0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000001FA798
<60be00>        Bytecode index = 55, inlineDepth = 3, PC offset = 0x000000000018D538
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000001F9EC0, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: jdk/internal/reflect/DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000001FA758
<60be00>        Bytecode index = 23, inlineDepth = 2, PC offset = 0x00000000001F9058
<60be00> JIT inline frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x00000000000B0850, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000000B1C70
<60be00>        Bytecode index = 104, inlineDepth = 1, PC offset = 0x000000000043BED8
<60be00> JIT frame: bp = 0x0000000000622C88, pc = 0x000003FF6FA8C726, unwindSP = 0x0000000000622B80, cp = 0x0000000000597740, arg0EA = 0x0000000000622C90, jitInfo = 0x000003FF6D0F2C78
<60be00>        Method: net/adoptopenjdk/loadTest/adaptors/MauveAdaptor.executeTest()Lnet/adoptopenjdk/loadTest/adaptors/AdaptorInterface$ResultStatus; !j9method 0x0000000000597D20
<60be00>        Bytecode index = 104, inlineDepth = 0, PC offset = 0x0000000000000E2C
<60be00>        stackMap=0x000003FF6D0F5074, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFF88)
<60be00>        Described JIT args starting at 0x0000000000622C90 for U16(0x0001) slots
<60be00>                O-Slot: : a0[0x0000000000622C90] = 0x000000003A18A568
<60be00>        Described JIT temps starting at 0x0000000000622C10 for IDATA(0x000000000000000F) slots
<60be00>                I-Slot: : t14[0x0000000000622C10] = 0x000000003C1DD598
<60be00>                I-Slot: : t13[0x0000000000622C18] = 0x0000000039A7D878
<60be00>                O-Slot: : t12[0x0000000000622C20] = 0x000000003C4EC468
<60be00>                O-Slot: : t11[0x0000000000622C28] = 0x00000000B046DBB8
<60be00>                I-Slot: : t10[0x0000000000622C30] = 0x0000000039A7D7F8
<60be00>                O-Slot: : t9[0x0000000000622C38] = 0x00000000B046DB60
<60be00>                I-Slot: : t8[0x0000000000622C40] = 0x0000000039F004D8
<60be00>                O-Slot: : t7[0x0000000000622C48] = 0x00000000B046D0B0
<60be00>                O-Slot: : t6[0x0000000000622C50] = 0x00000000B046DB60
<60be00>                I-Slot: : t5[0x0000000000622C58] = 0x000000003D716568
<60be00>                I-Slot: : t4[0x0000000000622C60] = 0x0000000039F00468
<60be00>                I-Slot: : t3[0x0000000000622C68] = 0x00000000B046DAA0
<60be00>                I-Slot: : t2[0x0000000000622C70] = 0x0000000039ADA6D8
<60be00>                I-Slot: : t1[0x0000000000622C78] = 0x00000000B046DAA0
<60be00>                I-Slot: : t0[0x0000000000622C80] = 0x0000000039F004D8
<60be00>        JIT-RegisterMap = UDATA(0x0000000000001C00)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AA0] = UDATA(0x000003FF6F9A7400) (jit_r6)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AA8] = UDATA(0x00000000B046DAA0) (jit_r7)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AB0] = UDATA(0x00000000B046D0B0) (jit_r8)
<60be00>                JIT-RegisterMap-I-Slot[0x0000000000622AB8] = UDATA(0x000000003C4EC4B8) (jit_r9)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622AC0] = 0x00000000B046DAA0 (jit_r10)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622980] = 0x000000003C4EC4B8 (jit_r11)
<60be00>                JIT-RegisterMap-O-Slot[0x0000000000622978] = 0x0000000039A7EF00 (jit_r12)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BB0] = UDATA(0x00000000B046C910) (jit_r6)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BB8] = UDATA(0x00000000B046CAB8) (jit_r7)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BC0] = UDATA(0x000000003C4CC980) (jit_r8)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BC8] = UDATA(0x000003FF6FA48600) (jit_r9)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BD0] = UDATA(0x000003FF6F748300) (jit_r10)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BD8] = UDATA(0x000003FF6F7482F4) (jit_r11)
<60be00>        JIT-Frame-RegisterMap[0x0000000000622BE0] = UDATA(0x0000000039A7EF00) (jit_r12)
<60be00> I2J values: PC = 0x000003FF8C550DA5, A0 = 0x0000000000622D18, walkSP = 0x0000000000622CA0, literals = 0x00000000005DE878, JIT PC = 0x000003FF8BCDD7C0, pcAddress = 0x0000000000622968, decomp = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622CB0, sp = 0x0000000000622CA0, pc = 0x000003FF8C550DA5, cp = 0x00000000005DE2C0, arg0EA = 0x0000000000622D18, flags = 0x0000000000000000
<60be00>        Method: net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V !j9method 0x00000000005DE878
<60be00>        Bytecode index = 129
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D18 for 0x000000000000000D slots
<60be00>                O-Slot: a0[0x0000000000622D18] = 0x000000003A012FA0
<60be00>                I-Slot: t1[0x0000000000622D10] = 0x00000000B49CD598
<60be00>                I-Slot: t2[0x0000000000622D08] = 0x00000000B49A7760
<60be00>                I-Slot: t3[0x0000000000622D00] = 0x0000000000000000
<60be00>                I-Slot: t4[0x0000000000622CF8] = 0x00000000005DE878
<60be00>                I-Slot: t5[0x0000000000622CF0] = 0x0000000000000000
<60be00>                O-Slot: t6[0x0000000000622CE8] = 0x000000003A012FC8
<60be00>                O-Slot: t7[0x0000000000622CE0] = 0x000000003A012FE0
<60be00>                O-Slot: t8[0x0000000000622CD8] = 0x000000003A18A568
<60be00>                I-Slot: t9[0x0000000000622CD0] = 0x0000000000000000
<60be00>                I-Slot: t10[0x0000000000622CC8] = 0x000000013A0101A8
<60be00>                I-Slot: t11[0x0000000000622CC0] = 0x000000000060BC88
<60be00>                I-Slot: t12[0x0000000000622CB8] = 0x000003FF3554DC6E
<60be00> Bytecode frame: bp = 0x0000000000622D30, sp = 0x0000000000622D20, pc = 0x000003FF35944416, cp = 0x0000000000518B10, arg0EA = 0x0000000000622D70, flags = 0x0000000000000000
<60be00>        Method: java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V !j9method 0x0000000000519920
<60be00>        Bytecode index = 94
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D70 for 0x0000000000000008 slots
<60be00>                O-Slot: a0[0x0000000000622D70] = 0x000000003A0123E0
<60be00>                O-Slot: a1[0x0000000000622D68] = 0x000000003A013000
<60be00>                O-Slot: t2[0x0000000000622D60] = 0x000000003A012EF0
<60be00>                O-Slot: t3[0x0000000000622D58] = 0x000000003A012FA0
<60be00>                I-Slot: t4[0x0000000000622D50] = 0x00000001B49A7788
<60be00>                I-Slot: t5[0x0000000000622D48] = 0x0000000000000000
<60be00>                I-Slot: t6[0x0000000000622D40] = 0x0000000000000000
<60be00>                I-Slot: t7[0x0000000000622D38] = 0x0000000000000000
<60be00> Bytecode frame: bp = 0x0000000000622D88, sp = 0x0000000000622D78, pc = 0x000003FF35A3B395, cp = 0x00000000005EAA00, arg0EA = 0x0000000000622D90, flags = 0x0000000000000000
<60be00>        Method: java/util/concurrent/ThreadPoolExecutor$Worker.run()V !j9method 0x00000000005EABC0
<60be00>        Bytecode index = 5
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622D90 for 0x0000000000000001 slots
<60be00>                I-Slot: a0[0x0000000000622D90] = 0x00000000B49A7788
<60be00> Bytecode frame: bp = 0x0000000000622DA8, sp = 0x0000000000622D98, pc = 0x000003FF354BB3E7, cp = 0x00000000000B8C90, arg0EA = 0x0000000000622DB8, flags = 0x0000000000000000
<60be00>        Method: java/lang/Thread.run()V !j9method 0x00000000000BA710
<60be00>        Bytecode index = 15
<60be00>        Using local mapper
<60be00>        Locals starting at 0x0000000000622DB8 for 0x0000000000000002 slots
<60be00>                I-Slot: a0[0x0000000000622DB8] = 0x00000000B49A77B8
<60be00>                I-Slot: t1[0x0000000000622DB0] = 0x00000000B49A7788
<60be00> JNI call-in frame: bp = 0x0000000000622DE0, sp = 0x0000000000622DC0, pc = 0x000003FF909AAC86, cp = 0x0000000000000000, arg0EA = 0x0000000000622DE0, flags = 0x0000000000000000
<60be00>        New ELS = 0x0000000000000000
<60be00> JNI native method frame: bp = 0x0000000000622E08, sp = 0x0000000000622DE8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x0000000000622E08, flags = 0x0000000000000000
<60be00> <end of stack>
<60be00> *** END STACK WALK (rc = NONE) ***

2) I went through the code changes listed at https://github.com/eclipse-openj9/openj9/compare/710fd952924c673ee702fcbc311c99d70095ecde...c19e4267798b13e6ee21c71290a2877c07fc4abb, in which the only PR that looks suspicious is https://github.com/eclipse-openj9/openj9/pull/15752 (which was to support Virtual Threads in JIT on zLinux) given the hang never happens on other platforms and never occurs with JIT off on zLinux.

So I will revert the changes to see how it goes in Grinder.

ChengJin01 commented 2 years ago

So far the problem can only be produced on the failing build at https://openj9-jenkins.osuosl.org/job/Grinder/1408/consoleText as I tried both with the latest build and with the reverted change at https://github.com/ChengJin01/openj9/tree/revert_jit_zlinux_thrd_pin_15752 in which case the hang never occurred. I think the JIT team still need to take a look to see what happened in there.

FYI: @r30shah

gacholio commented 2 years ago

The impression I get from @pshipton is that this happened exactly once and has never been reproduced, so I'm not sure there's any reason to blame any change. Was the pinning support updated in the next build?

ChengJin01 commented 2 years ago

The impression I get from @pshipton is that this happened exactly once and has never been reproduced

As I mentioned above, the hang is only reproduced on the failing build but never on the build right before the failing build or on the latest build.

Was the pinning support updated in the next build?

I didn't see any update on these two files in https://github.com/eclipse-openj9/openj9/pull/15752 or anything related to the pinning support after it was merged.

ChengJin01 commented 2 years ago

Just talked to GAC, we close the issue given there is no way to reproduce it in the latest build and no further evidence to prove the hang was caused by any change unless the problem shows up again.