search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.29k stars 723 forks source link

ClassLoadingTest_special_5m hang at 100% #17890

Open pshipton opened 1 year ago

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_0/82 ClassLoadingTest_special_5m_3 -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:count=0 -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_0/82/system_test_output.tar.gz

CLT 17:41:52.111 - Completed 100.0%. Number of tests started=24227 (+0)
CLT 17:42:12.072 - Completed 100.0%. Number of tests started=24227 (+0)
CLT 17:42:32.136 - Completed 100.0%. Number of tests started=24227 (+0)
STF 17:42:48.406 - Heartbeat: Process CLT is still running
CLT 17:42:52.122 - Completed 100.0%. Number of tests started=24227 (+0)
CLT 17:43:12.113 - Completed 100.0%. Number of tests started=24227 (+0)
CLT 17:43:30.085 - **POSSIBLE HANG DETECTED**
STF 17:43:30.584 - **FAILED** Process CLT has hung
        !stack 0x10024943400    !j9vmthread 0x10024943400       !j9thread 0x100249341c8 tid 0x5e902f5 (99156725) // (load-8)
        !stack 0x10025db8300    !j9vmthread 0x10025db8300       !j9thread 0x10024979970 tid 0x1ee043f (32375871) // (ClassByNameCache Reaper)
        !stack 0x10025c51700    !j9vmthread 0x10025c51700       !j9thread 0x10025c8f448 tid 0x24e0241 (38666817) // (ClassLoaderDeadlock-HelperThread)

> !stack 0x10024943400 (load-8)
<10024943400>   !j9method 0x0000010022849348   java/lang/Object.waitImpl(JI)V
<10024943400>   !j9method 0x0000010022849328   java/lang/Object.wait(JI)V
<10024943400>   !j9method 0x0000010022849308   java/lang/Object.wait(J)V
<10024943400>   !j9method 0x0000010022B58F80   java/lang/Thread.join(JI)V
<10024943400>   !j9method 0x0000010022B58F40   java/lang/Thread.join()V
<10024943400>   !j9method 0x0000010024EDA0C8   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.testDeadlock()V
<10024943400>   !j9method 0x0000010024EDA0A8   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.main([Ljava/lang/String;)V
<10024943400>   !j9method 0x0000010024EDA088   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.runTest()V
<10024943400>   !j9method 0x0000010024E640E8   jdk/internal/reflect/GeneratedMethodAccessor2.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<10024943400>   !j9method 0x00000100231F5868   jdk/internal/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<10024943400>   !j9method 0x0000010022B50B68   java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<10024943400>   !j9method 0x0000010024EDB120   net/adoptopenjdk/loadTest/adaptors/ArbitraryJavaAdaptor.executeTest()Lnet/adoptopenjdk/loadTest/adaptors/AdaptorInterface$ResultStatus;
<10024943400>   !j9method 0x0000010024F4B978   net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V
<10024943400>   !j9method 0x0000010024E04890   java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
<10024943400>   !j9method 0x0000010024F192B8   java/util/concurrent/ThreadPoolExecutor$Worker.run()V
<10024943400>   !j9method 0x0000010022B59000   java/lang/Thread.run()V
<10024943400>                           JNI call-in frame
<10024943400>                           Native method frame

> !stack 0x10025c51700 (ClassLoaderDeadlock-HelperThread)
<10025c51700>                           Native method frame
pshipton commented 11 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_4/104 - p8-java1-ibm01 ClassLoadingTest_special_5m_8 -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_4/104/system_test_output.tar.gz

  thread id: 46071843
   registers:
    cr     = 0x0000000000000000   ctr    = 0x0000000000000000   fpscr  = 0x0000000000000000   gpr0   = 0x0000000000000000
    gpr1   = 0x0000010028637a00   gpr10  = 0x0000000000000000   gpr11  = 0x0000000000000000   gpr12  = 0x0000000000000000
    gpr13  = 0x0000000000000000   gpr14  = 0x0000000000000000   gpr15  = 0x0000000000000000   gpr16  = 0x0000000000000000
    gpr17  = 0x0000000000000000   gpr18  = 0x0000000000000000   gpr19  = 0x0000000000000000   gpr2   = 0x0000000000000000
    gpr20  = 0x0000000000000000   gpr21  = 0x0000000000000000   gpr22  = 0x0000000000000000   gpr23  = 0x0000000000000000
    gpr24  = 0x0000000000000000   gpr25  = 0x0000000000000000   gpr26  = 0x0000000000000000   gpr27  = 0x0000000000000000
    gpr28  = 0x0000000000000000   gpr29  = 0x0000000000000000   gpr3   = 0x0000000000000000   gpr30  = 0x0000000000000000
    gpr31  = 0x0000000000000000   gpr4   = 0x0000000000000000   gpr5   = 0x0000000000000000   gpr6   = 0x0000000000000000
    gpr7   = 0x0000000000000000   gpr8   = 0x0000000000000000   gpr9   = 0x0000000000000000   iar    = 0x0000000000000000
    lr     = 0x0000000000000000   msr    = 0x0000000000000000   xer    = 0x0000000000000000
   native stack sections:
    0x10010000000 to 0x1002889b2e0 (length 0x1889b2e0)
   native stack frames:
   properties:
    current effective priority=0     current/last signal taken=0
    processor on which I'm bound=0   processor usage=ffff000a
    scheduling policy=52             suspend count=1
    thread flags=8410404             thread state=5
    type of thread wait=1            wait channel=0
   associated Java thread:
    name:          ClassLoaderDeadlock-HelperThread
    Thread object: java/lang/Thread @ 0xa00000001b75110
    Native info:   !j9vmthread 0x10026dde400  !stack 0x10026dde400
    Daemon:        false
    ID:            12777 (0x31e9)
    Priority:      5
    Thread.State:  WAITING
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
      waiting to be notified on: "Unknown monitor name" with ID 0x10026cf5058 owner name: <unowned>
    Java stack frames: <no frames to print>

https://github.com/adoptium/aqa-systemtest/blob/master/openjdk.test.classloading/src/test.classloading/net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.java#L146

pshipton commented 11 months ago

@gacholio any ideas why the ClassLoaderDeadlock-HelperThread is stuck?

gacholio commented 10 months ago

The stack of the helper thread is empty and the object still has its threadRef, so it looks like for some reason we're not calling the cleanup code (or we called it and it somehow failed).

The native stack of the stuck thread might be informative.

pshipton commented 10 months ago

There isn't much. If you want to poke around in gdb I can get you on the machine.

#0  0x09000000042e2b28 in genSystemCoreUsingGencore (portLibrary=0x1f9, filename=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/aix/omrosdump_helpers.c:203
#1  0x09000000042e1f28 in omrdump_create (portLibrary=0x9001000a07bf8d0 <_$STATIC_BSS+2880>,
    filename=0x9001000a0733968 <rasDumpPendingHooks+80> "\t", dumpType=0x10010128a10 "\t",
    userData=0x9001000a0733950 <rasDumpPendingHooks+56>)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/unix/omrosdump.c:118
#2  0x0900000004375430 in doSystemDump (agent=0x0, label=0x0, context=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/rasdump/dmpagent.c:763
#3  0x090000000437ddf4 in protectedDumpFunction (portLibrary=0x0, userData=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/rasdump/dmpagent.c:2897
#4  0x09000000042db1e0 in omrsig_protect (portLibrary=0x9001000a07bf8d0 <_$STATIC_BSS+2880>,
    fn=@0x9001000a0734580: 0x90000000437ddc0 <protectedDumpFunction>, fn_arg=0x1001011d508,
    handler=@0x9001000a0733950: 0x9000000043762c0 <signalHandler>, handler_arg=0x0, flags=505, result=0x1001011d4e8)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/unix/omrsignal.c:425
#5  0x090000000437da40 in runDumpAgent (vm=0x0, agent=0x0, context=0x0, state=0x2700000027,
    detail=0x9000000043d4efc <pushEventFrame+1852> "user", timeNow=<unknown type>)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/rasdump/dmpagent.c:2875
#6  0x09000000043cabe0 in triggerDumpAgents (vm=0x1001011dfd0, self=0x0, eventFlags=<unknown type>,
    eventData=<error reading variable>)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/rasdump/trigger.c:1041
#7  0x09000000062f882c in sigQuitHandler (portLibrary=0x0, userData=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/jcl/common/sigquit.c:122
#8  0x09000000042db1e0 in omrsig_protect (portLibrary=0x9001000a07bf8d0 <_$STATIC_BSS+2880>,
    fn=@0x9001000a0796be8: 0x9000000062f8740 <sigQuitHandler>, fn_arg=0x10010128a10,
    handler=@0x9001000a0ac4928: 0x900000003fd7180 <structuredSignalHandlerVM>, handler_arg=0x10010128a10, flags=506,
    result=0x1001011e580) at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/unix/omrsignal.c:425
#9  0x09000000062f85ac in sigQuitWrapper (portLibrary=0x0, gpType=0, gpInfo=0x1001011e680,
    userData=0x9000000043113c0 <omrthread_detach+28344>)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/openj9/runtime/jcl/common/sigquit.c:148
#10 0x09000000042dfd54 in asynchSignalReporter (userData=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/unix/omrsignal.c:824
#11 0x0900000004344590 in thread_wrapper (arg=0x0)
    at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/thread/common/omrthread.c:1733
#12 0x090000000051fe14 in ?? ()
#13 0x0000000000000000 in ?? ()
gacholio commented 10 months ago

That looks like the wrong thread - if it isn't then I'm not sure there's anything else I could glean from it.

The one we're interested in is:

        !stack 0x10025c51700    !j9vmthread 0x10025c51700       !j9thread 0x10025c8f448 tid 0x24e0241 (38666817) // (ClassLoaderDeadlock-HelperThread)
pshipton commented 10 months ago

Just the following, which shows up in info threads but not !threads.

(gdb) info threads
  Id   Target Id                              Frame
* 1    process 1                              0x09000000042e2b28 in genSystemCoreUsingGencore (portLibrary=0x1f9,
    filename=0x0) at /home/jenkins/workspace/Build_JDK17_ppc64_aix_Personal/omr/port/aix/omrosdump_helpers.c:203
  2    Thread 1 (tid 112984817, sleeping)     aix-thread: ptrace (52, 112984817) returned -1 (errno = 3 No such process)
  thread id: 112984817
   registers:
    cr     = 0x0000000000000000   ctr    = 0x0000000000000000   fpscr  = 0x0000000000000000   gpr0   = 0x000000000000000
0
    gpr1   = 0x0fffffffffffa0f0   gpr10  = 0x0000000000000000   gpr11  = 0x0000000000000000   gpr12  = 0x000000000000000
0
    gpr13  = 0x0000000000000000   gpr14  = 0x0000000000000000   gpr15  = 0x0000000000000000   gpr16  = 0x000000000000000
0
    gpr17  = 0x0000000000000000   gpr18  = 0x0000000000000000   gpr19  = 0x0000000000000000   gpr2   = 0x000000000000000
0
    gpr20  = 0x0000000000000000   gpr21  = 0x0000000000000000   gpr22  = 0x0000000000000000   gpr23  = 0x000000000000000
0
    gpr24  = 0x0000000000000000   gpr25  = 0x0000000000000000   gpr26  = 0x0000000000000000   gpr27  = 0x000000000000000
0
    gpr28  = 0x0000000000000000   gpr29  = 0x0000000000000000   gpr3   = 0x0000000000000000   gpr30  = 0x000000000000000
0
    gpr31  = 0x0000000000000000   gpr4   = 0x0000000000000000   gpr5   = 0x0000000000000000   gpr6   = 0x000000000000000
0
    gpr7   = 0x0000000000000000   gpr8   = 0x0000000000000000   gpr9   = 0x0000000000000000   iar    = 0x000000000000000
0
    lr     = 0x0000000000000000   msr    = 0x0000000000000000   xer    = 0x0000000000000000
   native stack sections:
    0xfffffffffffa000 to 0x1000000000000000 (length 0x6000)
   native stack frames:
   properties:
    current effective priority=0     current/last signal taken=0
    processor on which I'm bound=0   processor usage=ffff0008
    scheduling policy=3c             suspend count=1
    thread flags=8410404             thread state=5
    type of thread wait=1            wait channel=0
   associated Java thread: <no associated Java thread>
pshipton commented 9 months ago

https://openj9-jenkins..org/job/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_2/110 ClassLoadingTest_special_5m_28 -Xcompressedrefs -Xgcpolicy:gencon -Xshareclasses -Xjit

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_2/110/system_test_output.tar.gz

14:43:12  CLT 19:30:41.463 - Completed 100.0%. Number of tests started=58444 (+0)
14:43:33  CLT 19:31:01.448 - Completed 100.0%. Number of tests started=58444 (+0)
14:43:55  CLT 19:31:21.421 - Completed 100.0%. Number of tests started=58444 (+0)
14:44:13  CLT 19:31:41.394 - Completed 100.0%. Number of tests started=58444 (+0)
14:44:35  CLT 19:32:01.467 - Completed 100.0%. Number of tests started=58444 (+0)
14:44:53  CLT 19:32:21.438 - Completed 100.0%. Number of tests started=58444 (+0)
14:45:15  CLT 19:32:41.412 - Completed 100.0%. Number of tests started=58444 (+0)
14:45:30  STF 19:32:59.146 - Heartbeat: Process CLT is still running
14:45:32  CLT 19:33:01.384 - Completed 100.0%. Number of tests started=58444 (+0)
14:45:32  CLT 19:33:01.384 - **POSSIBLE HANG DETECTED**
14:45:32  STF 19:33:01.656 - **FAILED** Process CLT has hung
!threads
        !stack 0x30010700       !j9vmthread 0x30010700  !j9thread 0x100100b9da0 tid 0x661010f (107020559) // (main)
        !stack 0x30012c00       !j9vmthread 0x30012c00  !j9thread 0x100100ba7b0 tid 0x36c02eb (57410283) // (JIT Compilation Thread-000 Suspended)
        !stack 0x30015100       !j9vmthread 0x30015100  !j9thread 0x10021669a00 tid 0x361001b (56688667) // (JIT Compilation Thread-001 Suspended)
        !stack 0x30017600       !j9vmthread 0x30017600  !j9thread 0x10021669f08 tid 0x46b0143 (74121539) // (JIT Compilation Thread-002 Suspended)
        !stack 0x30019c00       !j9vmthread 0x30019c00  !j9thread 0x1002166a410 tid 0xa9022f (11076143) // (JIT Compilation Thread-003 Suspended)
        !stack 0x3001c100       !j9vmthread 0x3001c100  !j9thread 0x100219ee780 tid 0x4e8026f (82313839) // (JIT Compilation Thread-004)
        !stack 0x3001e600       !j9vmthread 0x3001e600  !j9thread 0x100219eec88 tid 0x483016b (75694443) // (JIT Compilation Thread-005 Suspended)
        !stack 0x30020b00       !j9vmthread 0x30020b00  !j9thread 0x100219ef190 tid 0x79c012d (127664429) // (JIT Compilation Thread-006 Suspended)
        !stack 0x30023000       !j9vmthread 0x30023000  !j9thread 0x10021d73500 tid 0x2c50195 (46465429) // (JIT Diagnostic Compilation Thread-007 Suspended)
        !stack 0x30025500       !j9vmthread 0x30025500  !j9thread 0x10021d73a08 tid 0x320125 (3277093) // (JIT-SamplerThread)
        !stack 0x30027a00       !j9vmthread 0x30027a00  !j9thread 0x10021d73f10 tid 0x26c0107 (40632583) // (IProfiler)
        !stack 0x30128900       !j9vmthread 0x30128900  !j9thread 0x10022f99220 tid 0x42a003b (69861435) // (Common-Cleaner)
        !stack 0x301c9b00       !j9vmthread 0x301c9b00  !j9thread 0x10022f99c30 tid 0x3302f7 (3343095) // (Finalizer thread)
        !stack 0x301cc000       !j9vmthread 0x301cc000  !j9thread 0x10023130960 tid 0x5203d9 (5374937) // (Concurrent Mark Helper)
        !stack 0x301ce500       !j9vmthread 0x301ce500  !j9thread 0x10023130e68 tid 0x39d0173 (60621171) // (GC Worker)
        !stack 0x301d0b00       !j9vmthread 0x301d0b00  !j9thread 0x10023131370 tid 0x3e901c7 (65601991) // (GC Worker)
        !stack 0x301d3000       !j9vmthread 0x301d3000  !j9thread 0x10023446be0 tid 0x3db016d (64684397) // (GC Worker)
        !stack 0x301d5500       !j9vmthread 0x301d5500  !j9thread 0x100234470e8 tid 0x4ed0189 (82641289) // (GC Worker)
        !stack 0x301d7a00       !j9vmthread 0x301d7a00  !j9thread 0x100234475f0 tid 0x27003c1 (40895425) // (GC Worker)
        !stack 0x301d9f00       !j9vmthread 0x301d9f00  !j9thread 0x1002364b700 tid 0x7d502a1 (131400353) // (GC Worker)
        !stack 0x301dc400       !j9vmthread 0x301dc400  !j9thread 0x1002364bc08 tid 0x5b80155 (95945045) // (GC Worker)
        !stack 0x301de900       !j9vmthread 0x301de900  !j9thread 0x1002364c110 tid 0xca0285 (13238917) // (GC Worker)
        !stack 0x301e0e00       !j9vmthread 0x301e0e00  !j9thread 0x10023850180 tid 0x594026f (93586031) // (GC Worker)
        !stack 0x301e3400       !j9vmthread 0x301e3400  !j9thread 0x10023850688 tid 0x5ac0243 (95158851) // (GC Worker)
        !stack 0x301e5900       !j9vmthread 0x301e5900  !j9thread 0x10023850b90 tid 0x3ea0201 (65667585) // (GC Worker)
        !stack 0x301e7e00       !j9vmthread 0x301e7e00  !j9thread 0x10023a54d40 tid 0x62c02af (103547567) // (GC Worker)
        !stack 0x301ea300       !j9vmthread 0x301ea300  !j9thread 0x10023a55248 tid 0x30b0379 (51053433) // (GC Worker)
        !stack 0x301ec800       !j9vmthread 0x301ec800  !j9thread 0x10023a55750 tid 0x4150025 (68485157) // (GC Worker)
        !stack 0x301eed00       !j9vmthread 0x301eed00  !j9thread 0x10023c5a7e0 tid 0x252009b (38928539) // (GC Worker)
        !stack 0x3000f400       !j9vmthread 0x3000f400  !j9thread 0x10023c5b1f0 tid 0x69e0089 (111018121) // (Attach API wait loop)
        !stack 0x301f9300       !j9vmthread 0x301f9300  !j9thread 0x10023c5ace8 tid 0x45402d1 (72614609) // (OperatingSystemMXBean notification dispatcher)
        !stack 0x302ee800       !j9vmthread 0x302ee800  !j9thread 0x10023d11d40 tid 0x17502dd (24445661) // (OperatingSystemMXBean notification dispatcher)
        !stack 0x305b1d00       !j9vmthread 0x305b1d00  !j9thread 0x1002420a750 tid 0x670139 (6750521) // (load-10)
        !stack 0x30614f00       !j9vmthread 0x30614f00  !j9thread 0x10024209570 tid 0x4b400c1 (78905537) // (ClassByNameCache Reaper)
        !stack 0x308ace00       !j9vmthread 0x308ace00  !j9thread 0x10024209068 tid 0x53600c7 (87425223) // (ClassLoaderDeadlock-HelperThread)
!stack 0x305b1d00 // (load-10)
<305b1d00>      !j9method 0x0000000030028A48   java/lang/Object.waitImpl(JI)V
<305b1d00>      !j9method 0x0000000030028A28   java/lang/Object.wait(JI)V
<305b1d00>      !j9method 0x0000000030028A08   java/lang/Object.wait(J)V
<305b1d00>      !j9method 0x000000003005F880   java/lang/Thread.join(JI)V
<305b1d00>      !j9method 0x000000003005F840   java/lang/Thread.join()V
<305b1d00>      !j9method 0x0000000030548BC8   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.testDeadlock()V
<305b1d00>      !j9method 0x0000000030548BA8   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.main([Ljava/lang/String;)V
<305b1d00>      !j9method 0x0000000030548B88   net/adoptopenjdk/test/classloading/deadlock/DeadlockTest.runTest()V
<305b1d00>      !j9method 0x000000003076CBE8   jdk/internal/reflect/GeneratedMethodAccessor3.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<305b1d00>      !j9method 0x00000000301352F0   jdk/internal/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<305b1d00>      !j9method 0x0000000030057368   java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<305b1d00>      !j9method 0x0000000030549C20   net/adoptopenjdk/loadTest/adaptors/ArbitraryJavaAdaptor.executeTest()Lnet/adoptopenjdk/loadTest/adaptors/AdaptorInterface$ResultStatus;
<305b1d00>      !j9method 0x0000000030579568   net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V
<305b1d00>      !j9method 0x00000000304CEAC0   java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
<305b1d00>      !j9method 0x0000000030576EC0   java/util/concurrent/ThreadPoolExecutor$Worker.run()V
<305b1d00>      !j9method 0x000000003005F900   java/lang/Thread.run()V
<305b1d00>                              JNI call-in frame
<305b1d00>                              Native method frame
!stack 0x308ace00 // (ClassLoaderDeadlock-HelperThread)
<308ace00>                              Native method frame
thread id: 87425223
   registers:
    cr     = 0x0000000000000000   ctr    = 0x0000000000000000   fpscr  = 0x0000000000000000   gpr0   = 0x0000000000000000
    gpr1   = 0x0000010029d3fa00   gpr10  = 0x0000000000000000   gpr11  = 0x0000000000000000   gpr12  = 0x0000000000000000
    gpr13  = 0x0000000000000000   gpr14  = 0x0000000000000000   gpr15  = 0x0000000000000000   gpr16  = 0x0000000000000000
    gpr17  = 0x0000000000000000   gpr18  = 0x0000000000000000   gpr19  = 0x0000000000000000   gpr2   = 0x0000000000000000
    gpr20  = 0x0000000000000000   gpr21  = 0x0000000000000000   gpr22  = 0x0000000000000000   gpr23  = 0x0000000000000000
    gpr24  = 0x0000000000000000   gpr25  = 0x0000000000000000   gpr26  = 0x0000000000000000   gpr27  = 0x0000000000000000
    gpr28  = 0x0000000000000000   gpr29  = 0x0000000000000000   gpr3   = 0x0000000000000000   gpr30  = 0x0000000000000000
    gpr31  = 0x0000000000000000   gpr4   = 0x0000000000000000   gpr5   = 0x0000000000000000   gpr6   = 0x0000000000000000
    gpr7   = 0x0000000000000000   gpr8   = 0x0000000000000000   gpr9   = 0x0000000000000000   iar    = 0x0000000000000000
    lr     = 0x0000000000000000   msr    = 0x0000000000000000   xer    = 0x0000000000000000
   native stack sections:
    0x10010000000 to 0x1002a2bafe0 (length 0x1a2bafe0)
   native stack frames:
   properties:
    current effective priority=0     current/last signal taken=0
    processor on which I'm bound=0   processor usage=ffff000d
    scheduling policy=52             suspend count=1
    thread flags=8410404             thread state=5
    type of thread wait=1            wait channel=0
   associated Java thread:
    name:          ClassLoaderDeadlock-HelperThread
    Thread object: java/lang/Thread @ 0x6225a7a08
    Native info:   !j9vmthread 0x308ace00  !stack 0x308ace00
    Daemon:        false
    ID:            15806 (0x3dbe)
    Priority:      5
    Thread.State:  WAITING
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
      waiting to be notified on: "Unknown monitor name" with ID 0x100256c2068 owner name: <unowned>
    Java stack frames: <no frames to print>
gacholio commented 9 months ago

This looks like a problem in the test code (thread waiting to be notified). I highly doubt we have a VM failure to notify (we would see this all the time).

pshipton commented 9 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64_aix_Release_testList_2/79 ClassLoadingTest_special_5m_28

pshipton commented 8 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_ppc64_aix_Personal_testList_3/105 ClassLoadingTest_special_5m_19 -Xcompressedrefs -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:count=0

pshipton commented 6 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_ppc64_aix_Personal_testList_2/37/ ClassLoadingTest_special_5m_19

pshipton commented 6 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_sanity.system_ppc64_aix_Nightly_testList_2/788/ ClassLoadingTest_5m_0

pshipton commented 2 days ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_ppc64_aix_Personal_testList_2/162 ClassLoadingTest_special_5m_19

  thread id: 7865057
   registers:
    cr     = 0x0000000000000000   ctr    = 0x0000000000000000   fpscr  = 0x0000000000000000   gpr0   = 0x0000000000000000
    gpr1   = 0x00000100266979f0   gpr10  = 0x0000000000000000   gpr11  = 0x0000000000000000   gpr12  = 0x0000000000000000
    gpr13  = 0x0000000000000000   gpr14  = 0x0000000000000000   gpr15  = 0x0000000000000000   gpr16  = 0x0000000000000000
    gpr17  = 0x0000000000000000   gpr18  = 0x0000000000000000   gpr19  = 0x0000000000000000   gpr2   = 0x0000000000000000
    gpr20  = 0x0000000000000000   gpr21  = 0x0000000000000000   gpr22  = 0x0000000000000000   gpr23  = 0x0000000000000000
    gpr24  = 0x0000000000000000   gpr25  = 0x0000000000000000   gpr26  = 0x0000000000000000   gpr27  = 0x0000000000000000
    gpr28  = 0x0000000000000000   gpr29  = 0x0000000000000000   gpr3   = 0x0000000000000000   gpr30  = 0x0000000000000000
    gpr31  = 0x0000000000000000   gpr4   = 0x0000000000000000   gpr5   = 0x0000000000000000   gpr6   = 0x0000000000000000
    gpr7   = 0x0000000000000000   gpr8   = 0x0000000000000000   gpr9   = 0x0000000000000000   iar    = 0x0000000000000000
    lr     = 0x0000000000000000   msr    = 0x0000000000000000   xer    = 0x0000000000000000
   native stack sections:
    0x10010000000 to 0x10026a4a3e0 (length 0x16a4a3e0)
   native stack frames:
   properties:
    current effective priority=0     current/last signal taken=0
    processor on which I'm bound=0   processor usage=ffff000b
    scheduling policy=52             suspend count=1
    thread flags=8410404             thread state=5
    type of thread wait=1            wait channel=0
   associated Java thread:
    name:          ClassLoaderDeadlock-HelperThread
    Thread object: java/lang/Thread @ 0x620773de8
    Native info:   !j9vmthread 0x30510000  !stack 0x30510000
    Daemon:        false
    ID:            3022 (0xbce)
    Priority:      5
    Thread.State:  WAITING
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
      waiting to be notified on: "Unknown monitor name" with ID 0x10024d802d0 owner name: <unowned>
    Java stack frames: <no frames to print>