LambdaLoadTest hang? in Runtime.gc()

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_ppc64_aix_Personal_testList_2/134 - p8-java1-ibm07 LambdaLoadTest_special_J9_5m_8 -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_special.system_ppc64_aix_Personal_testList_2/134/system_test_output.tar.gz **FAILED** Process LT has timed out

There are multiple load-* threads waiting in Runtime.gc().

    Java stack frames:
     bp: 0x00000100266d4ed0  method: void java/lang/Runtime.gc()  (Native Method)
      objects: 0xa00000000035850
     bp: 0x00000100266d4f98  method: void java/lang/System.gc()  source: System.java:394
      objects: <no objects in this frame>
     bp: 0x00000100266d5000  method: void net/adoptopenjdk/test/lambda/TestLambdaCapture.testCaptureReturnedLambda()  source: TestLambdaCapture.java:135
      objects: 0xa000001d81d38d0 0xa000000004268d8 0xa000001d81d3858 0xa000001d81d37a8

[pshipton]

@dmitripivkine

[github-actions[bot]]

Issue Number: 20559 Status: Open Recommended Components: comp:gc, comp:vm, comp:test Recommended Assignees: pshipton, jasonfengj9, llxia

[dmitripivkine]

This thread got request to execute SystemGC and trying to acquire Exclusive Access:

!stack 0x1002505b100    !j9vmthread 0x1002505b100   !j9thread 0x1002504b888 tid 0x980291 (9962129) // (load-4)
!j9vmthread 0x1002505b100 publicFlags=1020 privateFlags=0 inNative=0 // load-4

16:10:21.075646309 *0x1002505B100 j9vm.368             Event       Posted Halt request to 1 threads.
16:10:21.075647719  0x1002505B100 j9vm.369             Event       Waiting for 1 thread halt responses.
17:15:05.042548194 *0x100224E2A00 j9vm.243             Exit       <J9SignalAsyncEvent returning 0
17:15:05.042548489  0x100224E2A00 j9vm.242             Entry      >J9SignalAsyncEvent targetThread=0x10025099a00 handlerKey=2
17:15:05.042548731  0x100224E2A00 j9vm.243             Exit       <J9SignalAsyncEvent returning 0

However there is another thread has set Halt flag (0x01) but still keeping VM Access (0x20) and prevents taking Exclusive:

    !stack 0x10025099a00    !j9vmthread 0x10025099a00   !j9thread 0x100250898c8 tid 0x32402cf (52691663) // (load-10)
    !j9vmthread 0x10025099a00 publicFlags=21 privateFlags=0 inNative=0 // load-10

Looking to this threads is dbx:

$t47    run                          running  52691663   u   no   sys
(dbx) thread current 47
(dbx) where
---------
  $r0:0x090000000481148c  $stkp:0x00000100259009b0   $toc:0x08001000a01ca050
  $r3:0x0000010025099a00    $r4:0x000001001218ffcc    $r5:0x00000100121e2ea8
  $r6:0x0000000000000000    $r7:0x0000000000000000    $r8:0x0a000001d90a2778
  $r9:0x00000000004d2a10   $r10:0x0900000004461ca4   $r11:0x0000010022061aa8
 $r12:0x000001001218ffbc   $r13:0x000001002590a800   $r14:0x0000010025aa87b8
 $r15:0x0000010025099a00   $r16:0x0000010021fcb658   $r17:0xffffffffffffffff
 $r18:0x0000010025900bf0   $r19:0x420000842509b678   $r20:0x09000000056b62bc
 $r21:0x00000100224113e0   $r22:0x0000010025900c00   $r23:0x0000000000000001
 $r24:0x09000000056c0168   $r25:0x09001000a0760b90   $r26:0x0a000000004d2a10
 $r27:0x0a000001d90a2748   $r28:0x0a000001d90a2748   $r29:0x0a000000004d2a10
 $r30:0x0a0000000026e860   $r31:0x0a000001d90a2778
 $iar:0x090000000448d100   $msr:0x800000000000d032    $cr:0x0000000044000280
$link:0x0900000004461d60   $ctr:0x0900000004461ca4   $xer:0x0000000020000010
          Condition status = 0:g 1:g 5:e 6:l
        [unset $noflregs to view floating point registers]
        [unset $novregs to view vector registers]
        [unset $novsregs to view vector scalar registers]

0 old_slow_jitRetranslateMethod(0x10025099a00) at 0x90000000448d100

So, looks like thread in old_slow_jitRetranslateMethod() did not drop VM Access and caused the hang. @hzongaro FYI

[gacholio]

I have not looked at the code, but the usual cause of issues like this is a monitor - specifically that VM access should be treated as a monitor enter for ordering purposes.