DaaLoadTest_daa3_special_5m_18_FAILED: j9mm.107 ASSERTION FAILED at openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3934: ((false && (_copyForwardScheme->isObjectInEvacuateMemory(objectPtr))))

Failure link

From an internal build(p10rhel053):

java version "11.0.17" 2022-10-18
IBM Semeru Runtime Certified Edition 11.0.17+2 (build 11.0.17+2)
Eclipse OpenJ9 VM 11.0.17+2 (build master-1719d2e01, JRE 11 Linux ppc64le-64-Bit Compressed References 20220826_382 (JIT enabled, AOT enabled)
OpenJ9   - 1719d2e01
OMR      - 938f0686f
JCL      - 3dc2dd0048 based on jdk-11.0.17+2)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-08-27T03:57:52.093Z] variation: Mode557
[2022-08-27T03:57:52.093Z] JVM_OPTIONS:  -XX:+UseCompressedOops -Xgcpolicy:balanced -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:count=0 

[2022-08-27T04:00:20.685Z] DLT 00:00:19.109 - Completed 40.2%. Number of tests started=1942 (+321)
[2022-08-27T04:00:32.320Z] DLT stderr 04:00:30.697 0x204d00    j9mm.107    *   ** ASSERTION FAILED ** at ../../../../../openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3934: ((false && (_copyForwardScheme->isObjectInEvacuateMemory(objectPtr))))
[2022-08-27T04:00:32.320Z] DLT stderr 04:00:30.697 0x1dd900    j9mm.107    *   ** ASSERTION FAILED ** at ../../../../../openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3934: ((false && (_copyForwardScheme->isObjectInEvacuateMemory(objectPtr))))
[2022-08-27T04:00:32.320Z] DLT stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/08/27 00:00:30 - please wait.
[2022-08-27T04:00:32.320Z] DLT stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/08/27 00:00:30 - please wait.

[2022-08-27T04:00:32.716Z] STF 00:00:32.357 - Overall result: **FAILED**
[2022-08-27T04:00:32.716Z] 
[2022-08-27T04:00:32.716Z] DaaLoadTest_daa3_special_5m_18_FAILED

FYI @dmitripivkine

This assertion means there are objects in String Table considered to be not alive (by CopyForward) and not located in Evacuate memory. This should not happen. The reasons for this might be internal for GC (malfunction somewhere) as well as external for GC (missed barrier etc.). We are trying to get more information. There is one of problematic objects:

> !j9object 0x8a1f82b0
J9VMJavaLangString at 0x000000008A1F82B0 {
struct J9Class* clazz = !j9class 0xD1800 // java/lang/String
Object flags = 0x00000000;
[B value = !fj9object 0x8a1f82c0 (offset = 0) (java/lang/String)
B coder = 0x00000001 (offset = 4) (java/lang/String)
I hash = 0x0B877CB6 (offset = 8) (java/lang/String)
"jdk.internal.reflect.GeneratedMethodAccessor190"
}

Also please note there is bad (stale?) O-slot in JIT frame points mid-object (it points to data part of byte array actually):

Checking THREAD STACKS...  <gc check (1): from debugger: THREAD STACKS: slot af100(1423d8) -> 8cd57020: class pointer not in a class segment>

<af100> J2I frame: bp = 0x00000000001423C8, sp = 0x0000000000142308, pc = 0x00007FFF985D04D2, cp = 0x00000000005B5DC0, arg0EA = 0x00000000001424C0, flags = 0x0000000010000000
<af100>     Method: net/adoptopenjdk/loadTest/LoadTestRunner.run()J !j9method 0x00000000005B6428
<af100>     Bytecode index = 582
<af100>     Using debug local mapper
<af100>     Locals starting at 0x00000000001424C0 for 0x000000000000001F slots
<af100>         O-Slot: a0[0x00000000001424C0] = 0x000000008AF48D00
<af100>         O-Slot: t1[0x00000000001424B8] = 0x000000008AF48D50
<af100>         I-Slot: t2[0x00000000001424B0] = 0x0000000000000000
<af100>         I-Slot: t3[0x00000000001424A8] = 0x00000182DD7353D1
<af100>         I-Slot: t4[0x00000000001424A0] = 0x0000000000000000
<af100>         I-Slot: t5[0x0000000000142498] = 0x0000000000000006
<af100>         I-Slot: t6[0x0000000000142490] = 0x0000000000000000
<af100>         I-Slot: t7[0x0000000000142488] = 0x00000182DD73A1F1
<af100>         I-Slot: t8[0x0000000000142480] = 0x0000000000000000
<af100>         I-Slot: t9[0x0000000000142478] = 0x00000182DD7576B1
<af100>         I-Slot: t10[0x0000000000142470] = 0x0000000000000000
<af100>         I-Slot: t11[0x0000000000142468] = 0x0000000000000796
<af100>         O-Slot: t12[0x0000000000142460] = 0x000000008AF48D60
<af100>         O-Slot: t13[0x0000000000142458] = 0x000000008AF48D70
<af100>         O-Slot: t14[0x0000000000142450] = 0x000000008AF48D88
<af100>         O-Slot: t15[0x0000000000142448] = 0x000000008AF48DA0
<af100>         O-Slot: t16[0x0000000000142440] = 0x000000008A108980
<af100>         O-Slot: t17[0x0000000000142438] = 0x000000008A1089B0
<af100>         O-Slot: t18[0x0000000000142430] = 0x000000008AF48DB0
<af100>         O-Slot: t19[0x0000000000142428] = 0x000000008AF48F80
<af100>         I-Slot: t20[0x0000000000142420] = 0x000000000000001E
<af100>         I-Slot: t21[0x0000000000142418] = 0x0000000000000001
<af100>         I-Slot: t22[0x0000000000142410] = 0x0000000000000834
<af100>         I-Slot: t23[0x0000000000142408] = 0x000000000000001E
<af100>         I-Slot: t24[0x0000000000142400] = 0x00000182DD83115E
<af100>         I-Slot: t25[0x00000000001423F8] = 0x000000008ACE0808
<af100>         I-Slot: t26[0x00000000001423F0] = 0x0000000000000834
<af100>         I-Slot: t27[0x00000000001423E8] = 0x0000000000000000
<af100>         I-Slot: t28[0x00000000001423E0] = 0x0000000000000000
<af100>         O-Slot: t29[0x00000000001423D8] = 0x000000008CD57020 <------ pointer mid-object
<af100>         I-Slot: t30[0x00000000001423D0] = 0x000000008C31B5E8
<af100>     JIT-J2I-RegisterMap[0x00000000001423A8] = UDATA(0x0000000000000000) (jit_r16)
<af100>     JIT-J2I-RegisterMap[0x00000000001423A0] = UDATA(0xFFFFFFFFFFFFFFFF) (jit_r17)
<af100>     JIT-J2I-RegisterMap[0x0000000000142398] = UDATA(0x0000000000000000) (jit_r18)
<af100>     JIT-J2I-RegisterMap[0x0000000000142390] = UDATA(0x0000000000000000) (jit_r19)
<af100>     JIT-J2I-RegisterMap[0x0000000000142388] = UDATA(0x0000000000000000) (jit_r20)
<af100>     JIT-J2I-RegisterMap[0x0000000000142380] = UDATA(0x0000000000000000) (jit_r21)
<af100>     JIT-J2I-RegisterMap[0x0000000000142378] = UDATA(0x0000000000000000) (jit_r22)
<af100>     JIT-J2I-RegisterMap[0x0000000000142370] = UDATA(0x0000000000000000) (jit_r23)
<af100>     JIT-J2I-RegisterMap[0x0000000000142368] = UDATA(0x0000000000000000) (jit_r24)
<af100>     JIT-J2I-RegisterMap[0x0000000000142360] = UDATA(0x0000000000000000) (jit_r25)
<af100>     JIT-J2I-RegisterMap[0x0000000000142358] = UDATA(0x0000000000000000) (jit_r26)
<af100>     JIT-J2I-RegisterMap[0x0000000000142350] = UDATA(0x0000000000000000) (jit_r27)
<af100>     JIT-J2I-RegisterMap[0x0000000000142348] = UDATA(0x0000000000000000) (jit_r28)
<af100>     JIT-J2I-RegisterMap[0x0000000000142340] = UDATA(0x000000008AD17A18) (jit_r29)
<af100>     JIT-J2I-RegisterMap[0x0000000000142338] = UDATA(0x000000008ACDB858) (jit_r30)
<af100>     JIT-J2I-RegisterMap[0x0000000000142330] = UDATA(0x00000000005B6500) (jit_r31)

> !j9x 0x000000008CD57000,80,4
0x8CD57000 :  00009c09 00000000 00000000 00000000 [ ................ ]
0x8CD57010 :  000cdc00 0000000a 8cd57020 00000000 [ ........ p...... ]<---- byte array
0x8CD57020 :  008c0900 00000000 00000000 00000000 [ ................ ]<---- points here
0x8CD57030 :  000cdc00 0000000a 8cd57040 00000000 [ ........@p...... ]
0x8CD57040 :  008c0900 00000000 00000000 00000000 [ ................ ]
0x8CD57050 :  006dc600 00000008 00000000 00000000 [ ..m............. ]
0x8CD57060 :  000dcd00 00000001 8cd57070 00000000 [ ........pp...... ]
0x8CD57070 :  8cd57030 00000000 000dcd00 00000001 [ 0p.............. ]

> !j9object 0x8CD57010
!J9IndexableObject 0x000000008CD57010 {
    struct J9Class* clazz = !j9arrayclass 0xCDC00   // [B
    Object flags = 0x00000000;
    U_32 size = 0x0000000A;
    [0] =   0, 0x00
    [1] =   9, 0x09
    [2] = 140, 0x8c
    [3] =   0, 0x00
    [4] =   0, 0x00
    [5] =   0, 0x00
    [6] =   0, 0x00
    [7] =   0, 0x00
    [8] =   0, 0x00
    [9] =   0, 0x00
}

This is violation and required attention, adding JIT labels as well. @0xdaryl FYI

eclipse-openj9 / openj9