search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JDK19 java/lang/Thread/virtual/stress/PinALot.java#id0 IllegalMonitorStateException - RuntimeException: count = 653974 #16258

Closed JasonFengJ9 closed 1 year ago

JasonFengJ9 commented 2 years ago

Failure link

From an internal build(rhel8le-rt1-4):

openjdk version "19.0.1-beta" 2022-10-18
IBM Semeru Runtime Open Edition 19.0.1+10-202211030535 (build 19.0.1-beta+10-202211030535)
Eclipse OpenJ9 VM 19.0.1+10-202211030535 (build master-a31c79403, JRE 19 Linux ppc64le-64-Bit Compressed References 20221103_70 (JIT enabled, AOT enabled)
OpenJ9   - a31c79403
OMR      - fc60df565
JCL      - 165f4c2690 based on jdk-19.0.1+10)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-11-03T06:27:44.683Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode150
[2022-11-03T06:27:44.683Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops 

[2022-11-03T06:58:37.258Z] TEST: java/lang/Thread/virtual/stress/PinALot.java#id0

[2022-11-03T06:58:37.258Z] STDERR:
[2022-11-03T06:58:37.258Z] Exception in thread "" java.lang.IllegalMonitorStateException
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
[2022-11-03T06:58:37.258Z]  at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[2022-11-03T06:58:37.258Z]  at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[2022-11-03T06:58:37.258Z]  at PinALot.lambda$main$0(PinALot.java:60)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
[2022-11-03T06:58:37.258Z]  at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
[2022-11-03T06:58:37.258Z] java.lang.RuntimeException: count = 653974
[2022-11-03T06:58:37.258Z]  at PinALot.main(PinALot.java:73)
[2022-11-03T06:58:37.258Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:578)
[2022-11-03T06:58:37.258Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2022-11-03T06:58:37.258Z]  at java.base/java.lang.Thread.run(Thread.java:1573)
[2022-11-03T06:58:37.258Z] 
[2022-11-03T06:58:37.258Z] JavaTest Message: Test threw exception: java.lang.RuntimeException: count = 653974

[2022-11-03T06:58:37.259Z] TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: count = 653974
[2022-11-03T06:58:37.259Z] --------------------------------------------------
[2022-11-03T07:03:24.593Z] Test results: passed: 845; failed: 1
[2022-11-03T07:03:57.879Z] Report written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_0/aqa-tests/TKG/output_16674568639805/jdk_lang_0/report/html/report.html
[2022-11-03T07:03:57.879Z] Results written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_0/aqa-tests/TKG/output_16674568639805/jdk_lang_0/work
[2022-11-03T07:03:57.879Z] Error: Some tests failed or other problems occurred.
[2022-11-03T07:03:57.879Z] 
[2022-11-03T07:03:57.879Z] jdk_lang_0_FAILED

50x internal grinder - 18/100 failed

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/41/ jdk_lang_0 https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/43 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

21:51:36  Exception in thread "" java.lang.IllegalMonitorStateException
21:51:36    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
21:51:36    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
21:51:36    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
21:51:36    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
21:51:36    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
21:51:36    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
21:51:36    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
21:51:36    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
21:51:36    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
21:51:36    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
21:51:36    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
21:51:36    at PinALot.lambda$main$0(PinALot.java:60)
21:51:36    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
21:51:36    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
21:51:36    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
21:51:36  java.lang.RuntimeException: count = 851595
21:51:36    at PinALot.main(PinALot.java:73)
21:51:36    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
21:51:36    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
21:51:36    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
21:51:36    at java.base/java.lang.Thread.run(Thread.java:1573)
pshipton commented 2 years ago

@tajila @fengxue-IS fyi. I assume a recent regression, or a test recently unexcluded.

tajila commented 2 years ago

20x grinder on ppc: https://openj9-jenkins.osuosl.org/job/Grinder/1502/ The grinder failed 4 times, due to https://github.com/eclipse-openj9/openj9/issues/16259

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/44

20:57:16  STDERR:
20:57:16  Exception in thread "" java.lang.IllegalMonitorStateException
20:57:16    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
20:57:16    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
20:57:16    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
20:57:16    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
20:57:16    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
20:57:16    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
20:57:16    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
20:57:16    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
20:57:16    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
20:57:16    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
20:57:16    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
20:57:16    at PinALot.lambda$main$0(PinALot.java:60)
20:57:16    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
20:57:16    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
20:57:16    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
20:57:16  java.lang.RuntimeException: count = 768366
20:57:16    at PinALot.main(PinALot.java:73)
20:57:16    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
20:57:16    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
20:57:16    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
20:57:16    at java.base/java.lang.Thread.run(Thread.java:1573)
tajila commented 2 years ago

20x grinder on aarch64: https://openj9-jenkins.osuosl.org/job/Grinder/1503/

tajila commented 2 years ago

I ran a grinder internally with a minor modification to ReentrantLock and I cant reproduce the failure.

tajila commented 2 years ago

The issue occurs when the carrier thread is set as the owner of the lock instead of the virtual thread. Investigating why this occurs

dmitripivkine commented 2 years ago

The issue occurs when the carrier thread is set as the owner of the lock instead of the virtual thread. Investigating why this occurs

Interesting... crash in Scavenger https://github.com/eclipse-openj9/openj9/issues/16249 I am still investigating bad (stale?) object in Nursery might be taken from java/lang/VirtualThread->carrierThread

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/44 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

23:39:00  Exception in thread "" java.lang.IllegalMonitorStateException
23:39:00    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
23:39:00    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
23:39:00    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
23:39:00    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
23:39:00    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
23:39:00    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
23:39:00    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
23:39:00    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
23:39:00    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
23:39:00    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
23:39:00    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
23:39:00    at PinALot.lambda$main$0(PinALot.java:60)
23:39:00    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
23:39:00    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
23:39:00    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
23:39:00  java.lang.RuntimeException: count = 658946
23:39:00    at PinALot.main(PinALot.java:73)
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/44 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

tajila commented 2 years ago

Just posting output from my grinder here so I dont lose it.

Exception in thread "" java.lang.IllegalMonitorStateException: owner: Thread[#25,ForkJoinPool-1-worker-1,5,CarrierThreads] 
current: VirtualThread[#24]/runnable@ForkJoinPool-1-worker-1

    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:177)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:496)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
    at PinALot.lambda$main$0(PinALot.java:60)
    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
java.lang.RuntimeException: count = 495082
    at PinALot.main(PinALot.java:73)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    at java.base/java.lang.Thread.run(Thread.java:1573)
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/47 https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/45

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/46

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/49

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/50

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/51

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/51

tajila commented 2 years ago

@dmitripivkine I've confirmed that the issue here is the carrierThread is a VirtualThread instance

tajila commented 1 year ago

@pshipton have you ever seen this on x86? Im thinking there is a memory ordering issue here.

pshipton commented 1 year ago

@tajila every occurrence I've seen is recorded here, so no, not on x86. I only see alinux, plinux, and once on aix.

tajila commented 1 year ago

Another kind of failure

22:20:35  Exception in thread "" java.lang.RuntimeException: wrong threadobjectThread[#25,ForkJoinPool-1-worker-1,5,CarrierThreads]
22:20:35    at java.base/java.lang.VirtualThread.scheduleUnpark(VirtualThread.java:596)
22:20:35    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:534)
22:20:35    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
22:20:35    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
22:20:35    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
22:20:35    at PinALot.lambda$main$0(PinALot.java:60)
22:20:35    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
22:20:35    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
22:20:35    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)
22:20:35  java.lang.RuntimeException: count = 412599
22:20:35    at PinALot.main(PinALot.java:73)
22:20:35    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
22:20:35    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
22:20:35    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
22:20:35    at java.base/java.lang.Thread.run(Thread.java:1573)
tajila commented 1 year ago

so far no failures when -Xjit:disableInliningOfNatives is specified

tajila commented 1 year ago

@0xdaryl looks like disabling natinve inlining fixes this issue. Is there a way to determine if the result of Thread.currentThread() is being used as a constant value?

tajila commented 1 year ago

Also, I find it interesting that I never reproduced this on x86

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/55 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Nightly/55 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

pshipton commented 1 year ago

@tajila see the reproduction on x86 added to the previous comment.

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/56 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/59 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Nightly/58 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/63 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/60 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

0xdaryl commented 1 year ago

@tajila : just wondering how the investigation here became focused on Thread.currentThread(). I'm not saying that's not related, but I'm not seeing the connection from the analysis described above.

The JIT inlined implementation is the same as it always was: insert a load of the current thread object from the threadObject field of theJ9VMThread.

tajila commented 1 year ago

There are places in j.l.Thread/VirtualThread where the JCL switches from the virtualThread to the carrierThread. This is to avoid yielding twice when performing blocking operations. We are seeing that in some cases, we expect the carriedthread but currentThread returns a virtual thread.

In the failure below

22:20:35  Exception in thread "" java.lang.RuntimeException: wrong threadobjectThread[#25,ForkJoinPool-1-worker-1,5,CarrierThreads]
22:20:35    at java.base/java.lang.VirtualThread.scheduleUnpark(VirtualThread.java:596)
22:20:35    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:534)
22:20:35    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
22:20:35    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
22:20:35    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
22:20:35    at PinALot.lambda$main$0(PinALot.java:60)
22:20:35    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
22:20:35    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
22:20:35    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:171)

here is the code

        if (!future.isDone()) {
            Thread carrier = this.carrierThread;
            // need to switch to current platform thread to avoid nested parking
            carrier.setCurrentThread(carrier);
            try {
                if (Thread.currentThread() instanceof VirtualThread) {
                    throw new RuntimeException("wrong threadobject" + Thread.currentThread());
                }
                future.cancel(false);
            } finally {
                carrier.setCurrentThread(this);
            }
        }
fengxue-IS commented 1 year ago

@0xdaryl when JIT does the inline, I found a code path which set currentThread as ImmutableField, does this have an impact on how JIT retrieves the value?

TR_J9VM::inlineNativeCall() // VMJ9.cpp:7659
  J9::SymbolReferenceTable::findOrCreateCurrentThreadSymbolRef() // J9SymbolReferenceTable.cpp:1359
    sym->setImmutableField()
0xdaryl commented 1 year ago

Is this snippet of code:

        if (!future.isDone()) {
            Thread carrier = this.carrierThread;
            // need to switch to current platform thread to avoid nested parking
            carrier.setCurrentThread(carrier);
            try {
                if (Thread.currentThread() instanceof VirtualThread) {
                    throw new RuntimeException("wrong threadobject" + Thread.currentThread());
                }
                future.cancel(false);
            } finally {
                carrier.setCurrentThread(this);
            }
        }

taken from the private VirtualThread.cancel(Future<?> future) method? (if not, where is it from?). I don't see the instanceof check in the latest version. Just confirming that is something you added for debugging rather than me not looking at the latest version.

tajila commented 1 year ago

taken from the private VirtualThread.cancel(Future<?> future) method?

Yes that code snippet was added for debugging purposes

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/65 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/64 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/66 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/69 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_mac_Nightly/68 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

JasonFengJ9 commented 1 year ago

Internal JDK19 build(paix826)

openjdk version "19.0.1-beta" 2022-10-18
IBM Semeru Runtime Open Edition 19.0.1+10-202301110056 (build 19.0.1-beta+10-202301110056)
Eclipse OpenJ9 VM 19.0.1+10-202301110056 (build master-26019dda0, JRE 19 AIX ppc64-64-Bit Compressed References 20230110_19 (JIT enabled, AOT enabled)
OpenJ9   - 26019dda0
OMR      - d668e1624
JCL      - fc0cc674b0 based on jdk-19.0.1+10)

[2023-01-11T09:24:23.560Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
[2023-01-11T09:24:23.560Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops 

[2023-01-11T10:01:43.681Z] TEST: java/lang/Thread/virtual/stress/PinALot.java#id0

[2023-01-11T10:01:43.682Z] STDERR:
[2023-01-11T10:01:43.682Z] Exception in thread "" java.lang.IllegalMonitorStateException
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
[2023-01-11T10:01:43.682Z]  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
[2023-01-11T10:01:43.682Z]  at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
[2023-01-11T10:01:43.682Z]  at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
[2023-01-11T10:01:43.682Z]  at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
[2023-01-11T10:01:43.682Z]  at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[2023-01-11T10:01:43.709Z]  at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[2023-01-11T10:01:43.709Z]  at PinALot.lambda$main$0(PinALot.java:60)
[2023-01-11T10:01:43.709Z]  at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
[2023-01-11T10:01:43.709Z]  at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
[2023-01-11T10:01:43.709Z]  at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:170)
[2023-01-11T10:01:43.709Z] java.lang.RuntimeException: count = 883657
[2023-01-11T10:01:43.709Z]  at PinALot.main(PinALot.java:73)
[2023-01-11T10:01:43.709Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[2023-01-11T10:01:43.709Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:578)
[2023-01-11T10:01:43.709Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2023-01-11T10:01:43.709Z]  at java.base/java.lang.Thread.run(Thread.java:1573)
[2023-01-11T10:01:43.709Z] 
[2023-01-11T10:01:43.709Z] JavaTest Message: Test threw exception: java.lang.RuntimeException: count = 883657

[2023-01-11T10:08:37.833Z] jdk_lang_1_FAILED
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_OpenJDK19/12/ jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

08:40:15  STDOUT:
08:40:15  2023-01-20T13:37:15.571221Z => 232096
08:40:15  2023-01-20T13:37:16.618866Z => 529169
08:40:15  2023-01-20T13:37:17.619106Z => 718708
08:40:15  2023-01-20T13:37:18.500721Z => 905066
08:40:15  STDERR:
08:40:15  Exception in thread "" java.lang.IllegalMonitorStateException
08:40:15    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
08:40:15    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
08:40:15    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
08:40:15    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
08:40:15    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
08:40:15    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
08:40:15    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
08:40:15    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
08:40:15    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
08:40:15    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
08:40:15    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
08:40:15    at PinALot.lambda$main$0(PinALot.java:60)
08:40:15    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
08:40:15    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
08:40:15    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:170)
08:40:15  java.lang.RuntimeException: count = 905066
08:40:15    at PinALot.main(PinALot.java:73)

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/94 jdk_lang_0 java/lang/Thread/virtual/stress/PinALot.java#id0

21:27:23  STDERR:
21:27:23  Exception in thread "" java.lang.IllegalMonitorStateException
21:27:23    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
21:27:23    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
21:27:23    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
21:27:23    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
21:27:23    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
21:27:23    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
21:27:23    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
21:27:23    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
21:27:23    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
21:27:23    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
21:27:23    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
21:27:23    at PinALot.lambda$main$0(PinALot.java:60)
21:27:23    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
21:27:23    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
21:27:23    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:170)
21:27:23  java.lang.RuntimeException: count = 716586
21:27:23    at PinALot.main(PinALot.java:73)

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Nightly/94 jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

23:51:23  Exception in thread "" java.lang.IllegalMonitorStateException
23:51:23    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
23:51:23    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
23:51:23    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
23:51:23    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
23:51:23    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
23:51:23    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
23:51:23    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
23:51:23    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
23:51:23    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
23:51:23    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
23:51:23    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
23:51:23    at PinALot.lambda$main$0(PinALot.java:60)
23:51:23    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
23:51:23    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
23:51:23    at java.base/jdk.internal.vm.Continuation.execute(Continuation.java:170)
23:51:23  java.lang.RuntimeException: count = 478451
23:51:23    at PinALot.main(PinALot.java:73)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/96 jdk_lang_0 & jdk_lang_1 java/lang/Thread/virtual/stress/PinALot.java#id0

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/97

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/98

0xdaryl commented 1 year ago

Not reproducible yet on an x86-64 Linux system with the latest build after 300 runs. The test runs fast enough that I will grind away on it for a while.

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Nightly/107

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/108

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/109/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/113 https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/113/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/114 https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/114/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_linux_Nightly/115/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/117/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Release/1/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/120/

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_sanity.openjdk_x86-64_windows_Nightly/3/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/122/

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_sanity.openjdk_aarch64_linux_Nightly/6/

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_linux_Nightly/124/ https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_sanity.openjdk_x86-64_windows_Nightly/7/

This continues to happen frequently, I won't continue reporting all the failures.

pshipton commented 1 year ago

@0xdaryl sometimes these things are only reproducible if you run all of jdk_lang_1

0xdaryl commented 1 year ago

As this isn't a blocking issue and the fact that I won't have time to complete the investigation on this for 0.37 due to other commitments I propose that it be moved out.

pshipton commented 1 year ago

@0xdaryl why do you say this isn't a blocker? Everything in the 0.37 milestone plan is considered a blocker or we move it out after some discussion. This problem occurs frequently in the builds, see https://github.com/eclipse-openj9/openj9/issues/16258#issuecomment-1425028423. Is there something about this test that makes the problem non-important for a release?

0xdaryl commented 1 year ago

I was going by the presence (or absence) of the blocker label that we've used in the past to indicate blocking issues for a release. @a7ehuo has graciously agreed to pick up the investigation on this one as I am unable to give it enough of my cycles this month.

JasonFengJ9 commented 1 year ago

Also seen at JDK20 x86-64_windows

21:57:52  openjdk version "20-internal" 2023-03-21
21:57:52  OpenJDK Runtime Environment (build 20-internal-adhoc.****.buildjdk20x86-64windowsnightly)
21:57:52  Eclipse OpenJ9 VM (build master-9b39d6c0380, JRE 20 Windows Server 2012 R2 amd64-64-Bit Compressed References 20230306_9 (JIT enabled, AOT enabled)
21:57:52  OpenJ9   - 9b39d6c0380
21:57:52  OMR      - d5af8d16e93
21:57:52  JCL      - 91eb72cb238 based on jdk-20+36)

22:01:37  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode150
22:01:37  JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -

23:46:19  TEST: java/lang/Thread/virtual/ThreadAPI.java#default

23:46:20  STDERR:
23:46:20  Exception in thread "" java.lang.IllegalMonitorStateException
23:46:20    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:175)
23:46:20    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1059)
23:46:20    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
23:46:20    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
23:46:20    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
23:46:20    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
23:46:20    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:676)
23:46:20    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:608)
23:46:20    at java.base/java.lang.Access.parkVirtualThread(Access.java:516)
23:46:20    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
23:46:20    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
23:46:20    at ThreadAPI.lambda$testJoin33$37(ThreadAPI.java:765)
23:46:20    at java.base/java.lang.VirtualThread.run(VirtualThread.java:310)
23:46:20    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:182)
23:46:20    at java.base/jdk.internal.vm.Continuation.enter(Continuation.java:170)

23:46:32  jdk_lang_0_FAILED

Peter: Created a new PR for this failure https://github.com/eclipse-openj9/openj9/issues/16978

a7ehuo commented 1 year ago

I'm able to reproduce the exception with jdk_lang_1 on Power with default options by running the full test of jdk_lang_1. Using Grinder, FR ~1/20. Locally, FR ~1/30. Next, I'll look at what methods are compiled and see if I could narrow it down

a7ehuo commented 1 year ago

When I reproduced java.lang.IllegalMonitorStateException, the owner thread (tid=28) is a virtual thread and its carrier thread is ForkJoinPool-1-worker-1. The current thread (tid=29) is the carrier thread ForkJoinPool-1-worker-1 [1]. 

Exception in thread "" java.lang.IllegalMonitorStateException:
OwnerThread: VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1 VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1  threadID= 28 isVirtual=true state=RUNNABLE
CurrentThread: Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] ForkJoinPool-1-worker-1 threadID= 29 isVirtual=false state=RUNNABLE

@tajila, when ReentrantLock$Sync.tryRelease compares (getExclusiveOwnerThread() != Thread.currentThread()), does it expect the owner thread to be the carrier thread, or does it expect the current thread to be the virtual thread? Or does it expect getExclusiveOwnerThread() to return the carrier thread or does it expect Thread.currentThread() to return the virtual thread? Or rephrase it another way, does it expect both threads to be virtual thread, or expect both threads to be carrier thread (non-virtual)?

I looked at the carrier thread ForkJoinPool-1-worker-1 (j9vmthread 0x76614009b700) [2]. J9VMThread.threadObject (0x00007661C6C67960) has tid=0x1D (29) which is the current thread [3]. ReentrantLock$Sync.tryRelease does aload CurrentThread[#318 MethodMeta +168], which will load this carrier thread tid=29.

[1]

Exception in thread "" java.lang.IllegalMonitorStateException: OwnerThread: VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1 VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1  threadID= 28 isVirtual=true state=RUNNABLE || CurrentThread: Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] ForkJoinPool-1-worker-1 threadID= 29 isVirtual=false state=RUNNABLE
    at java.base/java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:179)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007)
    at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:499)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(ScheduledThreadPoolExecutor.java:1063)
    at java.base/java.util.concurrent.ThreadPoolExecutor.remove(ThreadPoolExecutor.java:1777)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(ScheduledThreadPoolExecutor.java:293)
    at java.base/java.lang.VirtualThread.cancel(VirtualThread.java:611)
    at java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:541)
    at java.base/java.lang.Access.parkVirtualThread(Access.java:502)
    at java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
    at PinALot.lambda$main$0(PinALot.java:60)
    at java.base/java.lang.VirtualThread.run(VirtualThread.java:292)
    at java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread.java:179)
    at java.base/jdk.internal.vm.Continuation.enter(Continuation.java:170)

[2]

!stack 0x76614009b700   !j9vmthread 0x76614009b700  !j9thread 0x76614002aab8    tid 0x259480 (2462848) // (ForkJoinPool-1-worker-1)

[3]

J9VMThread at 0x76614009b700 {
  Fields for J9VMThread:
    0x0: struct JNINativeInterface_* functions = !jninativeinterface_ 0x00007661ECF01B18
    0x8: struct J9JavaVM* javaVM = !j9javavm 0x00007661E801A130
    0x10: UDATA* arg0EA = !j9x 0x000076614009B4F0
    0x18: UDATA* bytecodes = !j9x 0x0000000000000000
    0x20: UDATA* sp = !j9x 0x000076614009B3E8
    0x28: U8* pc = !j9x 0x0000000000000006
    0x30: struct J9Method* literals = !j9method 0x0000000000000000
    0x38: UDATA jitStackFrameFlags = 0x0000000000000000 (0)
    0x40: struct J9Object* jitException = !j9object 0x0000000000000000
    0x48: struct J9Object* currentException = !j9object 0x0000000000000000
    0x50: UDATA* stackOverflowMark = !j9x 0xFFFFFFFFFFFFFFFF
    0x58: UDATA* stackOverflowMark2 = !j9x 0x000076614009AE70
    0x60: U8* heapAlloc = !j9x 0x0000000000000000
    0x68: U8* heapTop = !j9x 0x0000000000000000
    0x70: IDATA tlhPrefetchFTA = 0x0000000000000000 (0)
    0x78: U8* nonZeroHeapAlloc = !j9x 0x0000000000000000
    0x80: U8* nonZeroHeapTop = !j9x 0x0000000000000000
    0x88: IDATA nonZeroTlhPrefetchFTA = 0x0000000000000000 (0)
    0x90: struct J9ThreadMonitor* publicFlagsMutex = !j9threadmonitor 0x00007661400971B8
    0x98: UDATA publicFlags = 0x00000000000A0081 (655489)
    0xa0: UDATA compressObjectReferences = 0x0000000000000000 (0)
    0xa8: struct J9Object* threadObject = !j9object 0x00007661C6C67960 // jdk/internal/misc/CarrierThread //<==== MethodMeta +168
    0xb0: void* lowTenureAddress = !j9x 0x00007661C6AF0000
    0xb8: void* highTenureAddress = !j9x 0x00007661C70F0000
    0xc0: void* heapBaseForActiveCardTable = !j9x 0x0000000000000000
    0xc8: void* activeCardTableBase = !j9x 0x00007661C6060000
    0xd0: UDATA heapSizeForActiveCardTable = 0x0000000000000000 (0)
    0xd8: void* heapBaseForBarrierRange0 = !j9x 0x00007661C6AF0000
    0xe0: UDATA heapSizeForBarrierRange0 = 0x0000000000600000 (6291456)
    0xe8: UDATA* jniLocalReferences = !j9x 0x0000000000000000
    0xf0: UDATA tempSlot = 0x00007661ACEB5E74 (130161885011572)
    0xf8: void* jitReturnAddress = !j9x 0x00007661AD2A5F38
    0x100: void* floatTemp1 = !j9x 0x0000766100000000
    0x108: void* floatTemp2 = !j9x 0x00007661E81C5908
    0x110: void* floatTemp3 = !j9x 0x00007661AD4B98A0
    0x118: void* floatTemp4 = !j9x 0x0000000000000000
    0x120: UDATA returnValue = 0x0000000000000011 (17)
    0x128: UDATA returnValue2 = 0x0000000000000001 (1)
    0x130: UDATA* objectFlagSpinLockAddress = !j9x 0x0000000000000000
    0x138: struct J9JavaStack* stackObject = !j9javastack 0x0000766140099E70
    0x140: struct J9Thread* osThread = !j9thread 0x000076614002AAB8
    0x148: UDATA inspectionSuspendCount = 0x0000000000000000 (0)
    0x150: UDATA inspectorCount = 0x0000000000000000 (0)
    0x158: U32 eventFlags = 0x00000010 (16)
    0x15c: U32 osrFrameIndex = 0x00000000 (0)
    0x160: void* codertTOC = !j9x 0x0000000000000000
    0x168: U8* cardTableVirtualStart = !j9x 0x000076269522A880
    0x170: struct J9Object* stopThrowable = !j9object 0x0000000000000000
    0x178: struct J9Object* outOfMemoryError = !j9object 0x00007661C6C67A30 // java/lang/OutOfMemoryError
    0x180: UDATA* jniCurrentReference = !j9x 0x0000000000000000
    0x188: UDATA* jniLimitReference = !j9x 0x0000000000000000
    0x190: struct J9VMThread* linkNext = !j9vmthread 0x000076613000F400
    0x198: struct J9VMThread* linkPrevious = !j9vmthread 0x00007661E83C3D00
    0x1a0: UDATA privateFlags = 0x0000000000000002 (2)
    0x1a8: UDATA jitTOC = 0x00007661E7FBEE00 (130162875952640)
    0x1b0: UDATA ferReturnType = 0x0000000000000000 (0)
    0x1b8: U64 ferReturnValue = 0x0000000000000000 (0)
    0x1c0: U64 mgmtBlockedTimeTotal = 0x0000000000000000 (0)
    0x1c8: U64 mgmtBlockedTimeStart = 0x0000000000000000 (0)
    0x1d0: U64 mgmtWaitedTimeTotal = 0x0000000000000000 (0)
    0x1d8: U64 mgmtWaitedTimeStart = 0x0000000000000000 (0)
    0x1e0: UDATA jniVMAccessCount = 0x0000000000000000 (0)
    0x1e8: UDATA debugEventData1 = 0x0000000000000000 (0)
    0x1f0: UDATA debugEventData2 = 0x0000000000000000 (0)
    0x1f8: UDATA debugEventData3 = 0x0000000000000000 (0)
    0x200: UDATA debugEventData4 = 0x0000000000000000 (0)
    0x208: UDATA debugEventData5 = 0x0000000000000000 (0)
    0x210: UDATA debugEventData6 = 0x0000000000000000 (0)
    0x218: UDATA debugEventData7 = 0x0000000000000000 (0)
    0x220: UDATA debugEventData8 = 0x0000000000000000 (0)
    0x228: struct J9StackElement* classLoadingStack = !j9stackelement 0x0000000000000000
    0x230: UDATA jitTransitionJumpSlot = 0x0000000000000000 (0)
    0x238: struct J9ThreadMonitor* gcClassUnloadingMutex = !j9threadmonitor 0x0000000000000000
    0x240: struct J9VMThread* gcClassUnloadingThreadPrevious = !j9vmthread 0x0000000000000000
    0x248: struct J9VMThread* gcClassUnloadingThreadNext = !j9vmthread 0x0000000000000000
    0x250: struct J9StackWalkState* stackWalkState = !j9stackwalkstate 0x000076614009BA00
    0x258: struct J9VMEntryLocalStorage* entryLocalStorage = !j9vmentrylocalstorage 0x00007661A4E9C000
    0x260: UDATA gpProtected = 0x0000000000000001 (1)
    0x268: struct J9VMGCSublistFragment gcRememberedSet = !j9vmgcsublistfragment 0x000076614009B968
    0x298: struct MM_GCRememberedSetFragment sATBBarrierRememberedSetFragment = !mm_gcrememberedsetfragment 0x000076614009B998
    0x2c8: void* gcTaskListPtr = !j9x 0x0000000000000000
    0x2d0: UDATA* dropBP = !j9x 0x0000000000000000
    0x2d8: UDATA dropFlags = 0x0000000000000000 (0)
    0x2e0: struct J9Pool* monitorEnterRecordPool = !j9pool 0x0000766140046440
    0x2e8: struct J9MonitorEnterRecord* monitorEnterRecords = !j9monitorenterrecord 0x0000000000000000
    0x2f0: UDATA* jniArrayCache = !j9x 0x0000000000000000
    0x2f8: UDATA* jniArrayCache2 = !j9x 0x0000000000000000
    0x300: struct J9StackWalkState inlineStackWalkState = !j9stackwalkstate 0x000076614009BA00
    0x5b8: struct J9JITDecompilationInfo* decompilationStack = !j9jitdecompilationinfo 0x0000000000000000
    0x5c0: struct J9ModronThreadLocalHeap allocateThreadLocalHeap = !j9modronthreadlocalheap 0x000076614009BCC0
    0x5f0: struct J9ModronThreadLocalHeap nonZeroAllocateThreadLocalHeap = !j9modronthreadlocalheap 0x000076614009BCF0
    0x620: void* sidecarEvent = !j9x 0x0000000000000000
    0x628: struct PortlibPTBuffers_struct* ptBuffers = !portlibptbuffers_ 0x0000000000000000
    0x630: struct J9Object* blockingEnterObject = !j9object 0x00007661C6C50698 // java/util/concurrent/ForkJoinPool
    0x638: void* gcExtensions = !j9x 0x000076614009D488
    0x640: void* gpInfo = !j9x 0x0000000000000000
    0x648: void* jitVMwithThreadInfo = !j9x 0x0000766130004E10
    0x650: U8* profilingBufferEnd = !j9x 0x0000766140003360 // "!Ce���w�"
    0x658: U8* profilingBufferCursor = !j9x 0x000076614000301F // ""
    0x660: UDATA* j2iFrame = !j9x 0x0000000000000000
    0x668: UDATA currentOSStackFree = 0x000000000007C070 (508016)
    0x670: UDATA mgmtBlockedCount = 0x0000000000000003 (3)
    0x678: UDATA mgmtWaitedCount = 0x0000000000004C4E (19534)
    0x680: UDATA mgmtBlockedStart = 0x0000000000000000 (0)
    0x688: UDATA mgmtWaitedStart = 0x0000000000000000 (0)
    0x690: UDATA cardTableShiftSize = 0x0000000000000009 (9)
    0x698: void* aotVMwithThreadInfo = !j9x 0x0000000000000000
    0x6a0: UDATA asyncEventFlags = 0x0000000000000000 (0)
    0x6a8: struct J9Object* forceEarlyReturnObjectSlot = !j9object 0x0000000000000000
    0x6b0: struct J9MonitorEnterRecord* jniMonitorEnterRecords = !j9monitorenterrecord 0x0000000000000000
    0x6b8: struct J9DLTInformationBlock dltBlock = !j9dltinformationblock 0x000076614009BDB8
    0x878: struct J9VMGCSegregatedAllocationCacheEntry* segregatedAllocationCache = !j9vmgcsegregatedallocationcacheentry 0x0000000000000000
    0x880: struct J9StackWalkState* activeWalkState = !j9stackwalkstate 0x0000000000000000
    0x888: void* jniCalloutArgs = !j9x 0x0000000000000000
    0x890: struct J9VMThread* exclusiveVMAccessQueueNext = !j9vmthread 0x0000000000000000
    0x898: struct J9VMThread* exclusiveVMAccessQueuePrevious = !j9vmthread 0x0000000000000000
    0x8a0: struct J9Object* javaLangThreadLocalCache = !j9object 0x0000000000000000
    0x8a8: UDATA jitCountDelta = 0x0000000000000002 (2)
    0x8b0: UDATA maxProfilingCount = 0x0000000000001771 (6001)
    0x8b8: j9objectmonitor_t[] objectMonitorLookupCache = !j9x 0x000076614009BFB8
    0x9b8: UDATA jniCriticalCopyCount = 0x0000000000000000 (0)
    0x9c0: UDATA jniCriticalDirectCount = 0x0000000000000000 (0)
    0x9c8: struct J9Pool* jniReferenceFrames = !j9pool 0x0000000000000000
    0x9d0: U32 ludclInlineDepth = 0x00000000 (0)
    0x9d4: U32 ludclBPOffset = 0x00000000 (0)
    0x9d8: UDATA jitCurrentRIFlags = 0x0000000000000000 (0)
    0x9e0: UDATA jitPendingRIFlags = 0x0000000000000000 (0)
    0x9e8: struct J9RIParameters* riParameters = !j9riparameters 0x0000766140082260
    0x9f0: UDATA* osrJittedFrameCopy = !j9x 0x0000000000000000
    0x9f8: struct J9OSRBuffer* osrBuffer = !j9osrbuffer 0x0000000000000000
    0xa00: void* osrReturnAddress = !j9x 0x0000000000000000
    0xa08: void* osrScratchBuffer = !j9x 0x0000000000000000
    0xa10: void* jitArtifactSearchCache = !j9x 0x000076613000CA80
    0xa18: void* jitExceptionHandlerCache = !j9x 0x000076613001C6F0
    0xa20: void* jitPrivateData = !j9x 0x00007661A4FD7DC0
    0xa28: struct J9Method* jitMethodToBeCompiled = !j9method 0x0000000000000000
    0xa30: UDATA privateFlags2 = 0x0000000000000000 (0)
    0xa38: struct OMR_VMThread* omrVMThread = !omr_vmthread 0x000076614009C1D8
    0xa40: void* gpuInfo = !j9x 0x0000000000000000
    0xa48: void* startOfMemoryBlock = !j9x 0x000076614009B6C0
    0xa50: UDATA inNative = 0x0000000000000000 (0)
    0xa58: struct J9JITDecompilationInfo* lastDecompilation = !j9jitdecompilationinfo 0x0000000000000000
    0xa60: struct J9GSParameters gsParameters = !j9gsparameters 0x000076614009C160
    0xa70: UDATA readBarrierRangeCheckBase = 0xFFFFFFFFFFFFFFFF (-1)
    0xa78: UDATA readBarrierRangeCheckTop = 0x0000000000000000 (0)
    0xa80: U32 readBarrierRangeCheckBaseCompressed = 0xFFFFFFFF (4294967295)
    0xa84: U32 readBarrierRangeCheckTopCompressed = 0x00000000 (0)
    0xa88: UDATA safePointCount = 0x0000000000000000 (0)
    0xa90: volatile struct J9HashTable* utfCache = !j9hashtable 0x000076613000BF70
    0xa98: U64* ffiArgs = !j9x 0x0000000000000000
    0xaa0: UDATA ffiArgCount = 0x0000000000000000 (0)
    0xaa8: struct J9VMContinuation* currentContinuation = !j9vmcontinuation 0x0000000000000000
    0xab0: UDATA continuationPinCount = 0x0000000000000000 (0)
    0xab8: UDATA ownedMonitorCount = 0x0000000000000000 (0)
    0xac0: UDATA callOutCount = 0x0000000000000001 (1)
    0xac8: struct J9Object* carrierThreadObject = !j9object 0x00007661C6C67960 // jdk/internal/misc/CarrierThread
    0xad0: struct J9Object* scopedValueCache = !j9object 0x0000000000000000
}
> !j9object 0x00007661C6C67960
!J9Object 0x00007661C6C67960 {
    struct J9Class* clazz = !j9class 0x7661E83C9700 // jdk/internal/misc/CarrierThread
    Object flags = 0x0000002A;
    J lockword = 0x00007661E83C3D08 (offset = 0) (java/lang/Object) <hidden>
    J eetop = 0x000076614009B700 (offset = 8) (java/lang/Thread)
    J tid = 0x000000000000001D (offset = 16) (java/lang/Thread) // <--- threadID=29
    Ljava/lang/String; name = !fj9object 0x7661c6c67b98 (offset = 40) (java/lang/Thread)
    Z interrupted = 0x00000000 (offset = 152) (java/lang/Thread)
    Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x7661c6b5bd38 (offset = 48) (java/lang/Thread)
    Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x7661c6c67bb8 (offset = 56) (java/lang/Thread)
    Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x7661c6c67c08 (offset = 64) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 72) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 80) (java/lang/Thread)
    Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 88) (java/lang/Thread)
    Ljava/lang/Object; interruptLock = !fj9object 0x7661c6c67c40 (offset = 96) (java/lang/Thread)
    Ljava/lang/Object; parkBlocker = !fj9object 0x7661c6c50698 (offset = 104) (java/lang/Thread)
    Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 112) (java/lang/Thread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 120) (java/lang/Thread)
    Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x7661c6c67c58 (offset = 128) (java/lang/Thread)
    J threadLocalRandomSeed = 0x9D7C942D058D2357 (offset = 24) (java/lang/Thread)
    I threadLocalRandomProbe = 0x3C6EF372 (offset = 156) (java/lang/Thread)
    I threadLocalRandomSecondarySeed = 0x00000000 (offset = 160) (java/lang/Thread)
    Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x7661c6c67c68 (offset = 136) (java/lang/Thread)
    Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 144) (java/lang/Thread)
    Z started = 0x00000001 (offset = 164) (java/lang/Thread)
    Z stopCalled = 0x00000000 (offset = 168) (java/lang/Thread)
    J tls = 0x0000000000000000 (offset = 32) (java/lang/Thread) <hidden>
    Ljava/util/concurrent/ForkJoinPool; pool = !fj9object 0x7661c6c50698 (offset = 176) (java/util/concurrent/ForkJoinWorkerThread)
    Ljava/util/concurrent/ForkJoinPool$WorkQueue; workQueue = !fj9object 0x7661c6c67ae0 (offset = 184) (java/util/concurrent/ForkJoinWorkerThread)
    Z blocking = 0x00000000 (offset = 172) (jdk/internal/misc/CarrierThread)
}
tajila commented 1 year ago

@tajila, when ReentrantLock$Sync.tryRelease compares (getExclusiveOwnerThread() != Thread.currentThread()), does it expect the owner thread to be the carrier thread, or does it expect the current thread to be the virtual thread?

The expectation is that getExclusiveOwnerThread() is a carrier thread, also that Thread.currentThread() is also a carrier thread. So both threads are expected to be carrierThreads.

I think you are seeing the same thing I saw where after the exception is thrown the core file indicates that a carrier thread is the only sensible result, however the getExclusiveOwnerThread() != Thread.currentThread() test fails which is inexplicable to me.

a7ehuo commented 1 year ago

Two things I found from the core when the IllegalMonitorStateException happened:

detail "java/lang/IllegalMonitorStateException", exception "OwnerThread: VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1 VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1  threadID= 28 isVirtual=true state=RUNNABLE || CurrentThread: Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] ForkJoinPool-1-worker-1 threadID= 29 isVirtual=false state=RUNNABLE"

  1. On the Java call stack leading up to the exception, the methods from VirtualThread.cancel to AbstractQueuedSynchronizer.release are inlined into VirtualThread.parkNanos. ReentrantLock$Sync.tryRelease that throws IllegalMonitorStateException is not inlined into any method [1].

  2. Thread.currentThread() is carrier threadForkJoinPool-1-worker-1 (0x7d9190095900). It's threadObject is 0x00007D920C19D750. Its tid is 0x1D(29) [2]. The unwindSP for ReentrantLock$Sync.tryRelease is 0x00007D918000B1B0 [3]. Based on the objects stored on the stack [4], it looks to me the sync object for ReentrantLock is java/util/concurrent/locks/ReentrantLock$NonfairSync (0x00007d920c1ab398). its exclusiveOwnerThread is 0x7d920c19d648 [5]. 0x7d920c19d648 has tid as 0x1C (28) which is a virtual thread.

@tajila ReentrantLock$Sync.tryRelease just loads CurrentThread (n20n) and the exclusiveOwnerThread from ReentrantLock$Sync (n283n) and compares the two loaded thread objects (n25n) [6]. So far I don't see any issue with ReentrantLock$Sync.tryRelease. Since exclusiveOwnerThread is expected to be the carrier thread, which code is in charge of updating the java/util/concurrent/locks/AbstractOwnableSynchronizer.exclusiveOwnerThread from a virtual thread to a carrier thread for the ReentrantLock$Sync object? Maybe the investigation could be looked at from that perspective?

[1]

...
<7d9190095900> JIT frame: bp = 0x00007D918000B1F8, pc = 0x00007D91F2EE6B48, unwindSP = 0x00007D918000B1B0, cp = 0x00007D922C207AD0, arg0EA = 0x00007D918000B208, jitInfo = 0x00007D91F0D99378
<7d9190095900>  Method: java/util/concurrent/locks/ReentrantLock$Sync.tryRelease(I)Z !j9method 0x00007D922C1C0028
<7d9190095900>  Bytecode index = 194, inlineDepth = 0, PC offset = 0x00000000000003D0
<7d9190095900>  stackMap=0x00007D91F0D99926, slots=I16(0x0002) parmBaseOffset=I16(0x0010), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFF0)
<7d9190095900>  Described JIT args starting at 0x00007D918000B208 for U16(0x0001) slots
<7d9190095900>      I-Slot: : a0[0x00007D918000B208] = 0x00007D920C19D750
<7d9190095900>  Described JIT temps starting at 0x00007D918000B1E8 for IDATA(0x0000000000000002) slots
<7d9190095900>      O-Slot: : t1[0x00007D918000B1E8] = 0x00007D920C19D750
<7d9190095900>      O-Slot: : t0[0x00007D918000B1F0] = 0x00007D920C19D648
<7d9190095900>  JIT-RegisterMap = UDATA(0x0000000010000000)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB938] = UDATA(0x00007D922BFB5118) (jit_r31)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB930] = UDATA(0x00007D922BFB5158) (jit_r30)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB928] = UDATA(0x00007D922C3C0500) (jit_r29)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB920] = UDATA(0x0000000000000001) (jit_r28)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB918] = UDATA(0x00007D920C19D648) (jit_r27)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB910] = UDATA(0x00007D920C19DAB0) (jit_r26)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB908] = UDATA(0x00007D920C19D648) (jit_r25)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB900] = UDATA(0x0000000000000000) (jit_r24)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8F8] = UDATA(0x000512C38DF3831E) (jit_r23)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8F0] = UDATA(0x0000000000000000) (jit_r22)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8E8] = UDATA(0x0000000000000000) (jit_r21)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8E0] = UDATA(0x0000000000000001) (jit_r20)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8D8] = UDATA(0x00007D920C19D750) (jit_r19)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8D0] = UDATA(0x00007D920C19A720) (jit_r18)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8C8] = UDATA(0x00007D922BFB4FE0) (jit_r17)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8C0] = UDATA(0x00007D91F0D60038) (jit_r16)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8B8] = UDATA(0x0000000000000000) (jit_r15)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8B0] = UDATA(0x0000000000000000) (jit_r14)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8A8] = UDATA(0x00007D91F2BB68F0) (jit_r13)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8A0] = UDATA(0x0000000028884228) (jit_r12)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB898] = UDATA(0x00007D9231F195E0) (jit_r11)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB890] = UDATA(0x00007D91F2EE6B3C) (jit_r10)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB888] = UDATA(0x00007D91900D6568) (jit_r9)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB880] = UDATA(0x0000000000000080) (jit_r8)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB878] = UDATA(0x00007D91F0D60038) (jit_r7)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB870] = UDATA(0x00007D922BFB4FE0) (jit_r6)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB868] = UDATA(0x00007D918000B1B8) (jit_r5)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB860] = UDATA(0x000000000000000E) (jit_r4)
<7d9190095900>      JIT-RegisterMap-O-Slot[0x00007D91F2BAB858] = 0x00007D922BFB5118 (jit_r3)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB850] = UDATA(0x00007D9232FE6D00) (jit_r2)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB848] = UDATA(0x0000000000000000) (jit_r1)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB840] = UDATA(0x00007D91F2EE6B3C) (jit_r0)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8C0] = UDATA(0x00007D91F0D60038) (jit_r16)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8C8] = UDATA(0x00007D922BFB4FE0) (jit_r17)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8D0] = UDATA(0x00007D920C19A720) (jit_r18)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8D8] = UDATA(0x00007D920C19D750) (jit_r19)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8E0] = UDATA(0x0000000000000001) (jit_r20)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8E8] = UDATA(0x0000000000000000) (jit_r21)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8F0] = UDATA(0x0000000000000000) (jit_r22)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8F8] = UDATA(0x000512C38DF3831E) (jit_r23)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB900] = UDATA(0x0000000000000000) (jit_r24)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB908] = UDATA(0x00007D920C19D648) (jit_r25)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB910] = UDATA(0x00007D920C19DAB0) (jit_r26)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB918] = UDATA(0x00007D920C19D648) (jit_r27)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1C8] = UDATA(0x00007D920C19DA78) (jit_r28)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1D0] = UDATA(0x00007D920C1AB398) (jit_r29)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1D8] = UDATA(0x00007D922C212700) (jit_r30)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1E0] = UDATA(0x00007D920C1AB398) (jit_r31)
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C20EC30, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/locks/AbstractQueuedSynchronizer.release(I)Z !j9method 0x00007D922C20F4B0
<7d9190095900>  Bytecode index = 2, inlineDepth = 6, PC offset = 0x0000000000A19630
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1BCC50, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/locks/ReentrantLock.unlock()V !j9method 0x00007D922C1BD008
<7d9190095900>  Bytecode index = 5, inlineDepth = 5, PC offset = 0x00000000004234AC
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900788F0, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(Ljava/lang/Object;)Z !j9method 0x00007D9190078108
<7d9190095900>  Bytecode index = 104, inlineDepth = 4, PC offset = 0x0000000000016B30
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900730C0, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ThreadPoolExecutor.remove(Ljava/lang/Runnable;)Z !j9method 0x00007D9190074290
<7d9190095900>  Bytecode index = 7, inlineDepth = 3, PC offset = 0x00000000002221B0
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900A1A40, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(Z)Z !j9method 0x00007D91900A1D30
<7d9190095900>  Bytecode index = 32, inlineDepth = 2, PC offset = 0x00000000000185B0
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1CCC10, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/lang/VirtualThread.cancel(Ljava/util/concurrent/Future;)V !j9method 0x00007D922C1CDF40
<7d9190095900>  Bytecode index = 23, inlineDepth = 1, PC offset = 0x000000000001E5B0
<7d9190095900> JIT frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1CCC10, arg0EA = 0x00007D918000B6C0, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/lang/VirtualThread.parkNanos(J)V !j9method 0x00007D922C1CDEE0
<7d9190095900>  Bytecode index = 23, inlineDepth = 0, PC offset = 0x0000000000002730
<7d9190095900>  stackMap=0x00007D91F0DB61FC, slots=I16(0x0003) parmBaseOffset=I16(0x0018), parmSlots=U16(0x0001), localBaseOffset=I16(0xFD48)
<7d9190095900> JIT frame: bp = 0x00007D918000B1F8, pc = 0x00007D91F2EE6B48, unwindSP = 0x00007D918000B1B0, cp = 0x00007D922C207AD0, arg0EA = 0x00007D918000B208, jitInfo = 0x00007D91F0D99378
<7d9190095900>  Method: java/util/concurrent/locks/ReentrantLock$Sync.tryRelease(I)Z !j9method 0x00007D922C1C0028
<7d9190095900>  Bytecode index = 194, inlineDepth = 0, PC offset = 0x00000000000003D0
<7d9190095900>  stackMap=0x00007D91F0D99926, slots=I16(0x0002) parmBaseOffset=I16(0x0010), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFF0)
<7d9190095900>  Described JIT args starting at 0x00007D918000B208 for U16(0x0001) slots
<7d9190095900>      I-Slot: : a0[0x00007D918000B208] = 0x00007D920C19D750
<7d9190095900>  Described JIT temps starting at 0x00007D918000B1E8 for IDATA(0x0000000000000002) slots
<7d9190095900>      O-Slot: : t1[0x00007D918000B1E8] = 0x00007D920C19D750
<7d9190095900>      O-Slot: : t0[0x00007D918000B1F0] = 0x00007D920C19D648
<7d9190095900>  JIT-RegisterMap = UDATA(0x0000000010000000)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB938] = UDATA(0x00007D922BFB5118) (jit_r31)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB930] = UDATA(0x00007D922BFB5158) (jit_r30)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB928] = UDATA(0x00007D922C3C0500) (jit_r29)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB920] = UDATA(0x0000000000000001) (jit_r28)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB918] = UDATA(0x00007D920C19D648) (jit_r27)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB910] = UDATA(0x00007D920C19DAB0) (jit_r26)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB908] = UDATA(0x00007D920C19D648) (jit_r25)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB900] = UDATA(0x0000000000000000) (jit_r24)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8F8] = UDATA(0x000512C38DF3831E) (jit_r23)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8F0] = UDATA(0x0000000000000000) (jit_r22)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8E8] = UDATA(0x0000000000000000) (jit_r21)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8E0] = UDATA(0x0000000000000001) (jit_r20)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8D8] = UDATA(0x00007D920C19D750) (jit_r19)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8D0] = UDATA(0x00007D920C19A720) (jit_r18)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8C8] = UDATA(0x00007D922BFB4FE0) (jit_r17)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8C0] = UDATA(0x00007D91F0D60038) (jit_r16)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8B8] = UDATA(0x0000000000000000) (jit_r15)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8B0] = UDATA(0x0000000000000000) (jit_r14)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8A8] = UDATA(0x00007D91F2BB68F0) (jit_r13)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB8A0] = UDATA(0x0000000028884228) (jit_r12)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB898] = UDATA(0x00007D9231F195E0) (jit_r11)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB890] = UDATA(0x00007D91F2EE6B3C) (jit_r10)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB888] = UDATA(0x00007D91900D6568) (jit_r9)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB880] = UDATA(0x0000000000000080) (jit_r8)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB878] = UDATA(0x00007D91F0D60038) (jit_r7)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB870] = UDATA(0x00007D922BFB4FE0) (jit_r6)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB868] = UDATA(0x00007D918000B1B8) (jit_r5)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB860] = UDATA(0x000000000000000E) (jit_r4)
<7d9190095900>      JIT-RegisterMap-O-Slot[0x00007D91F2BAB858] = 0x00007D922BFB5118 (jit_r3)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB850] = UDATA(0x00007D9232FE6D00) (jit_r2)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB848] = UDATA(0x0000000000000000) (jit_r1)
<7d9190095900>      JIT-RegisterMap-I-Slot[0x00007D91F2BAB840] = UDATA(0x00007D91F2EE6B3C) (jit_r0)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8C0] = UDATA(0x00007D91F0D60038) (jit_r16)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8C8] = UDATA(0x00007D922BFB4FE0) (jit_r17)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8D0] = UDATA(0x00007D920C19A720) (jit_r18)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8D8] = UDATA(0x00007D920C19D750) (jit_r19)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8E0] = UDATA(0x0000000000000001) (jit_r20)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8E8] = UDATA(0x0000000000000000) (jit_r21)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8F0] = UDATA(0x0000000000000000) (jit_r22)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB8F8] = UDATA(0x000512C38DF3831E) (jit_r23)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB900] = UDATA(0x0000000000000000) (jit_r24)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB908] = UDATA(0x00007D920C19D648) (jit_r25)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB910] = UDATA(0x00007D920C19DAB0) (jit_r26)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D91F2BAB918] = UDATA(0x00007D920C19D648) (jit_r27)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1C8] = UDATA(0x00007D920C19DA78) (jit_r28)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1D0] = UDATA(0x00007D920C1AB398) (jit_r29)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1D8] = UDATA(0x00007D922C212700) (jit_r30)
<7d9190095900>  JIT-Frame-RegisterMap[0x00007D918000B1E0] = UDATA(0x00007D920C1AB398) (jit_r31)
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C20EC30, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/locks/AbstractQueuedSynchronizer.release(I)Z !j9method 0x00007D922C20F4B0
<7d9190095900>  Bytecode index = 2, inlineDepth = 6, PC offset = 0x0000000000A19630
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1BCC50, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/locks/ReentrantLock.unlock()V !j9method 0x00007D922C1BD008
<7d9190095900>  Bytecode index = 5, inlineDepth = 5, PC offset = 0x00000000004234AC
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900788F0, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.remove(Ljava/lang/Object;)Z !j9method 0x00007D9190078108
<7d9190095900>  Bytecode index = 104, inlineDepth = 4, PC offset = 0x0000000000016B30
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900730C0, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ThreadPoolExecutor.remove(Ljava/lang/Runnable;)Z !j9method 0x00007D9190074290
<7d9190095900>  Bytecode index = 7, inlineDepth = 3, PC offset = 0x00000000002221B0
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D91900A1A40, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask.cancel(Z)Z !j9method 0x00007D91900A1D30
<7d9190095900>  Bytecode index = 32, inlineDepth = 2, PC offset = 0x00000000000185B0
<7d9190095900> JIT inline frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1CCC10, arg0EA = 0x0000000000000000, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/lang/VirtualThread.cancel(Ljava/util/concurrent/Future;)V !j9method 0x00007D922C1CDF40
<7d9190095900>  Bytecode index = 23, inlineDepth = 1, PC offset = 0x000000000001E5B0
<7d9190095900> JIT frame: bp = 0x00007D918000B6A8, pc = 0x00007D91F36E35A8, unwindSP = 0x00007D918000B200, cp = 0x00007D922C1CCC10, arg0EA = 0x00007D918000B6C0, jitInfo = 0x00007D91F0DB1D78
<7d9190095900>  Method: java/lang/VirtualThread.parkNanos(J)V !j9method 0x00007D922C1CDEE0
<7d9190095900>  Bytecode index = 23, inlineDepth = 0, PC offset = 0x0000000000002730
<7d9190095900>  stackMap=0x00007D91F0DB61FC, slots=I16(0x0003) parmBaseOffset=I16(0x0018), parmSlots=U16(0x0001), localBaseOffset=I16(0xFD48)
...

[2]

    !stack 0x7d9190095900   !j9vmthread 0x7d9190095900  !j9thread 0x7d9190027c18    tid 0x2e7091 (3043473) // (ForkJoinPool-1-worker-1)
> !j9vmthread 0x7d9190095900
J9VMThread at 0x7d9190095900 {
  Fields for J9VMThread:
    0x0: struct JNINativeInterface_* functions = !jninativeinterface_ 0x00007D9232FE1B18
    0x8: struct J9JavaVM* javaVM = !j9javavm 0x00007D922C01A1D0
    0x10: UDATA* arg0EA = !j9x 0x00007D918000B1A8
        ... 
    0xa8: struct J9Object* threadObject = !j9object 0x00007D920C19D750 // jdk/internal/misc/CarrierThread
        ... 
> !j9object 0x00007D920C19D750 
!J9Object 0x00007D920C19D750 {
    struct J9Class* clazz = !j9class 0x7D922C3C0500 // jdk/internal/misc/CarrierThread
    Object flags = 0x0000002A;
    J lockword = 0x00007D922C3C3708 (offset = 0) (java/lang/Object) <hidden>
    J eetop = 0x00007D9190095900 (offset = 8) (java/lang/Thread)
    J tid = 0x000000000000001D (offset = 16) (java/lang/Thread)
        ...

[3]

<7d9190095900> JIT frame: bp = 0x00007D918000B1F8, pc = 0x00007D91F2EE6B48, unwindSP = 0x00007D918000B1B0, cp = 0x00007D922C207AD0, arg0EA = 0x00007D918000B208, jitInfo = 0x00007D91F0D99378
<7d9190095900>  Method: java/util/concurrent/locks/ReentrantLock$Sync.tryRelease(I)Z !j9method 0x00007D922C1C0028
....

[4]

(kca) whatis (0x00007D918000B1B0)/32
0x7d918000b1b0: 0x00007d922bfb6410 Obj - {java/lang/String} "OwnerThread: VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1 VirtualThread[#28]/runnable@ForkJoinPool-1-worker-1  threadID= 28 isVirtual=true state=RUNNABLE || CurrentThread: Thread[#29,ForkJoinPool-1-worker-1,5,CarrierThreads] Thread[#29,ForkJoinPool..."
0x7d918000b1b8: 0x00007d922bfb5118 Obj - {java/lang/IllegalMonitorStateException}
0x7d918000b1c0: 0x00007d922bfb5158 Obj - {java/lang/StringBuilder}
0x7d918000b1c8: 0x00007d920c19da78 Obj - {java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue}T
0x7d918000b1d0: 0x00007d920c1ab398 Obj - {java/util/concurrent/locks/ReentrantLock$NonfairSync}T
0x7d918000b1d8: 0x00007d922c212700 J9Class - {java/util/concurrent/locks/ReentrantLock$NonfairSync}
0x7d918000b1e0: 0x00007d920c1ab398 Obj - {java/util/concurrent/locks/ReentrantLock$NonfairSync}T
0x7d918000b1e8: 0x00007d920c19d750 Obj - {jdk/internal/misc/CarrierThread}T
0x7d918000b1f0: 0x00007d920c19d648 Obj - {java/lang/VirtualThread}T
0x7d918000b1f8: 0x00007d91f36e35a8 {java/lang/VirtualThread.parkNanos} +2508
0x7d918000b200: 0x00007d920c19d750 Obj - {jdk/internal/misc/CarrierThread}T
0x7d918000b208: 0x00007d920c19d750 Obj - {jdk/internal/misc/CarrierThread}T
0x7d918000b210: 0x00007d920c19d750 Obj - {jdk/internal/misc/CarrierThread}T
0x7d918000b218: 0x00007d922bfb4d68 Obj - {java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask}
0x7d918000b220: 0x00007d920c19da78 Obj - {java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue}T
0x7d918000b228: 0x00007d922bfb4d68 Obj - {java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask}
0x7d918000b230: 0x00007d922bfb4fb0 Obj - {java/util/concurrent/locks/AbstractQueuedSynchronizer$ExclusiveNode}
0x7d918000b238: 0x00007d920c1ab398 Obj - {java/util/concurrent/locks/ReentrantLock$NonfairSync}T
0x7d918000b240: 0xffffffffffffffff Unknown!
0x7d918000b248: 0x0000000000000000 Unknown!
...

[5]

> !j9object 0x00007d920c1ab398
!J9Object 0x00007D920C1AB398 {
    struct J9Class* clazz = !j9class 0x7D922C212700 // java/util/concurrent/locks/ReentrantLock$NonfairSync
    Object flags = 0x00000010;
    J lockword = 0x0000000000000008 (offset = 0) (java/lang/Object) <hidden>
    Ljava/lang/Thread; exclusiveOwnerThread = !fj9object 0x7d920c19d648 (offset = 16) (java/util/concurrent/locks/AbstractOwnableSynchronizer)
    J ownableSynchronizerLink = 0x00007D920C19D238 (offset = 8) (java/util/concurrent/locks/AbstractOwnableSynchronizer) <hidden>
    Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node; head = !fj9object 0x7d922bf9adc0 (offset = 24) (java/util/concurrent/locks/AbstractQueuedSynchronizer)
    Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node; tail = !fj9object 0x7d922bf9adf8 (offset = 32) (java/util/concurrent/locks/AbstractQueuedSynchronizer)
    I state = 0x00000001 (offset = 40) (java/util/concurrent/locks/AbstractQueuedSynchronizer)
}

> !fj9object 0x7d920c19d648 
!J9Object 0x00007D920C19D648 {
    struct J9Class* clazz = !j9class 0x7D922C1CE700 // java/lang/VirtualThread
    Object flags = 0x0000002A;
    J lockword = 0x0000000000000008 (offset = 0) (java/lang/Object) <hidden>
    J eetop = 0x0000000000000000 (offset = 8) (java/lang/Thread)
    J tid = 0x000000000000001C (offset = 16) (java/lang/Thread)
    Ljava/lang/String; name = !fj9object 0x7d920c00b360 (offset = 40) (java/lang/Thread)
    Z interrupted = 0x00000000 (offset = 152) (java/lang/Thread)
    Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x7d920c070cf8 (offset = 48) (java/lang/Thread)
    Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x7d920c071008 (offset = 56) (java/lang/Thread)
    Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 64) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 72) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 80) (java/lang/Thread)
    Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 88) (java/lang/Thread)
    Ljava/lang/Object; interruptLock = !fj9object 0x7d920c1aabb0 (offset = 96) (java/lang/Thread)
    Ljava/lang/Object; parkBlocker = !fj9object 0x0 (offset = 104) (java/lang/Thread)
    Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 112) (java/lang/Thread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 120) (java/lang/Thread)
    Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 128) (java/lang/Thread)
    J threadLocalRandomSeed = 0x0000000000000000 (offset = 24) (java/lang/Thread)
    I threadLocalRandomProbe = 0x00000000 (offset = 156) (java/lang/Thread)
    I threadLocalRandomSecondarySeed = 0x00000000 (offset = 160) (java/lang/Thread)
    Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x7d920c1994a0 (offset = 136) (java/lang/Thread)
    Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 144) (java/lang/Thread)
    Z started = 0x00000000 (offset = 164) (java/lang/Thread)
    Z stopCalled = 0x00000000 (offset = 168) (java/lang/Thread)
    J tls = 0x0000000000000000 (offset = 32) (java/lang/Thread) <hidden>
    Ljava/util/concurrent/Executor; scheduler = !fj9object 0x7d920c19a698 (offset = 184) (java/lang/VirtualThread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x7d920c19d860 (offset = 192) (java/lang/VirtualThread)
    Ljava/lang/Runnable; runContinuation = !fj9object 0x7d920c1aabc0 (offset = 200) (java/lang/VirtualThread)
    I state = 0x00000004 (offset = 172) (java/lang/VirtualThread)
    Z parkPermit = 0x00000000 (offset = 240) (java/lang/VirtualThread)
    Ljava/lang/Thread; carrierThread = !fj9object 0x7d920c19d750 (offset = 208) (java/lang/VirtualThread)
    Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x7d920c1aabd8 (offset = 216) (java/lang/VirtualThread)
    Ljava/lang/VirtualThread; linkNext = !fj9object 0x7d920c19db20 (offset = 232) (java/lang/VirtualThread) <hidden>
    Ljava/lang/VirtualThread; linkPrevious = !fj9object 0x7d920c19db20 (offset = 224) (java/lang/VirtualThread) <hidden>
    J inspectorCount = 0x0000000000000000 (offset = 176) (java/lang/VirtualThread) <hidden>
    I isSuspendedByJVMTI = 0x00000000 (offset = 244) (java/lang/VirtualThread) <hidden>
}

[6]

n9n       BBStart <block_2> (freq 83)                                                         [    0x7d917a003d40] bci=[0,0,532] rc=0 vc=855 vn=- li=-2 udi=- nc=1
n420n       GlRegDeps                                                                         [    0x7d917a097300] bci=[0,0,532] rc=1 vc=855 vn=- li=-1 udi=- nc=2
n421n         aRegLoad gr29   this<'this' parm Ljava/util/concurrent/locks/ReentrantLock$Sync;>[#396  Parm] [flags 0x40000107 0x0 ] (X!=0 X>=0 SeenRealReference )  [    0x7d917a097350] bci=[-1,0,173] rc=6 vc=855 vn=- li=-6 udi=- nc=0 flg=0x8104
...
...
n295n     aRegStore gr2                                                                       [    0x7d917a094bf0] bci=[1,1,84] rc=0 vc=855 vn=- li=-2 udi=- nc=1
n283n       aloadi  java/util/concurrent/locks/AbstractOwnableSynchronizer.exclusiveOwnerThread Ljava/lang/Thread;[#465  Shadow +24] [flags 0x80607 0x0 ]  [    0x7d917a094830] bci=[1,1,84] rc=2 vc=855 vn=- li=-2 udi=- nc=1
n421n         ==>aRegLoad
n289n     BBEnd </block_15> =====                                                             [    0x7d917a094a10] bci=[-1,8,174] rc=0 vc=855 vn=- li=-2 udi=- nc=1
n430n       GlRegDeps ()                                                                      [    0x7d917a097620] bci=[-1,8,174] rc=1 vc=855 vn=- li=-1 udi=- nc=3 flg=0x20
n428n         PassThrough gr2                                                                 [    0x7d917a097580] bci=[1,1,84] rc=1 vc=855 vn=- li=-1 udi=- nc=1
n283n           ==>aloadi
...
...
n288n     BBStart <block_19> (freq 9827)                                                      [    0x7d917a0949c0] bci=[-1,8,174] rc=0 vc=855 vn=- li=-1 udi=- nc=1
n431n       GlRegDeps ()                                                                      [    0x7d917a097670] bci=[-1,8,174] rc=1 vc=855 vn=- li=- udi=- nc=3 flg=0x20
n432n         aRegLoad gr2   <temp slot 6>[#467  Auto] [flags 0x4007 0x0 ] (SeenRealReference )  [    0x7d917a0976c0] bci=[-1,8,174] rc=2 vc=855 vn=- li=- udi=- nc=0 flg=0x8000
...
n25n      ifacmpne --> block_3 BBStart at n23n ()                                             [    0x7d917a004240] bci=[-1,14,174] rc=0 vc=855 vn=- li=-1 udi=- nc=3 flg=0x20
n432n       ==>aRegLoad
n20n        aload  CurrentThread[#318  MethodMeta +168] [flags 0x207 0x400 ]                  [    0x7d917a0040b0] bci=[-1,11,174] rc=1 vc=855 vn=- li=- udi=- nc=0
n437n       GlRegDeps ()                                                                      [    0x7d917a097850] bci=[-1,14,174] rc=1 vc=855 vn=- li=- udi=- nc=1 flg=0x20
n434n         ==>aRegLoad
tajila commented 1 year ago

@a7ehuo Your assessment is correct. The problem isnt in ReentrantLock$Sync.tryRelease the problem happens much earlier in when the lock owner is set. As you'll see in the code it should never be a VirtualThread.

If you look at some of my earlier comments you'll see the following example:

        if (!future.isDone()) {
            Thread carrier = this.carrierThread;
            // need to switch to current platform thread to avoid nested parking
            carrier.setCurrentThread(carrier);
            try {
                if (Thread.currentThread() instanceof VirtualThread) {
                    throw new RuntimeException("wrong threadobject" + Thread.currentThread());
                }
                future.cancel(false);
            } finally {
                carrier.setCurrentThread(this);
            }
        }

I modified the code to double check that it is not a VritualThread, however, I noticed that this would intermittently fail.

So the possibilities are: 1) The carrierThread isnt a carrierThread. However, if this is the case how come its always a carrier thread when we look at the core file 2) Thread.currentThread() is being cached 3) Similar to 2) Operations are being done out of order assuming that Thread.currentThread() is constant

tajila commented 1 year ago

Perhaps there are other possibilities but thats all the comes to mind at the moment

tajila commented 1 year ago

@a7ehuo Are you able to reproduce the issue with compilation of Thread.currentThread() disabled?

a7ehuo commented 1 year ago

Thanks so much for the clarification @tajila. I'll try excluding Thread.currentThread() and see how it goes. If the issue happens before ReentrantLock$Sync.tryRelease, I suspect it might be related to how methods are inlined into VirtualThread.parkNanos

a7ehuo commented 1 year ago

I'm able to reproduce the issue by excluding Thread.currentThread (Grinder 31731)

Grinder: 31731:
-Xjit:exclude={java/lang/Thread.currentThread*}

I haven't been able to repro the issue after adding more logging (Grinder 31728) as below. Locally I've tried over 500x without reproducing it with required logging. I'm still trying to repro the issue with the required logging in order to investigate it from JIT's perspective 

Grinder 31728: 
-Xnoaot -Xjit:{java/lang/VirtualThread.parkNanos*}(traceFull,traceILGen,traceCG,traceInlining,debugInliner,tracePRE,log=jitlog) -Xdump:system+java+snap+jit:events=throw,filter=java/lang/IllegalMonitorStateException#java/base/java/util/concurrent/locks/ReentrantLock*,request=exclusive+prepwalk+serial+preempt
mstoodle commented 1 year ago

Hi @a7ehuo did you also specify dontInline={java/lang/Thread.currentThread*} in those jgrinder runs? I could be wrong as it's been a while, but I'm not sure if exclude={} will prevent inlining of those methods. It may not matter (hard to imagine the aliasing isn't set up properly to ensure setCurrentThread blocks reuse of the current thread) but I'd rather say something stupid than not point out something that might matter.

a7ehuo commented 1 year ago

did you also specify dontInline={java/lang/Thread.currentThread*} in those jgrinder runs?

Thanks for pointing it out! I didn't add dontInline={java/lang/Thread.currentThread*}. I'll add that along exclude={..} and see how it goes

mstoodle commented 1 year ago

along those same lines, you may want to use a filter that excludes setCurrentThread() as well as currentThread() and validate that dontInline actually does prevent inlining for those relatively simple functions :) . Though I suppose we'd be a lot less likely to move a store around than a load. So maybe the first part isn't useful but I would make sure dontInline covers what might be recognized methods (probably does, but if you're going to the trouble to re-test....).