search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.23k stars 711 forks source link

abbs crash vmState=0x0005ff0b #19102

Open pshipton opened 4 months ago

pshipton commented 4 months ago

Internal build [zOS S390] 80 Load_Level_2.abbs.5mins.Mode103 - fyrec809 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs vmState [0x5ff0b]: {J9VMSTATE_JIT} {ProcessRelocations}

50x grinder passed

j> 07:05:43 Type=Segmentation error vmState=0x0005ff0b
j> 07:05:43 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000035
j> 07:05:43 Handler1=1995A498 Handler2=1A35C080
j> 07:05:43 gpr0=FF7FFFF9 gpr1=19F234F8 gpr2=4C511320 gpr3=47E83E84
j> 07:05:43 gpr4=3D5FBFE0 gpr5=1BDEA8F8 gpr6=0000005C gpr7=000A0000
j> 07:05:43 gpr8=8000004F gpr9=47E84588 gpr10=00000710 gpr11=4B52DDFC
j> 07:05:43 gpr12=199894B8 gpr13=4B52DEB4 gpr14=00000000 gpr15=4A719B48
j> 07:05:43 hgpr0=00000000 hgpr1=00000000 hgpr2=00000000 hgpr3=00000000
j> 07:05:43 hgpr4=00000000 hgpr5=00000000 hgpr6=00000000 hgpr7=00000000
j> 07:05:43 hgpr8=180F5BE8 hgpr9=00000000 hgpr10=00000000 hgpr11=00000069
j> 07:05:43 hgpr12=00000000 hgpr13=00000000 hgpr14=00000000 hgpr15=00000000
j> 07:05:43 fpc=00880000 psw0=078D1400 psw1=9B98BC44 sp=3D5FBFE0
j> 07:05:43 bea=1B98BC4C
j> 07:05:43 fpr0=0000000000000000 fpr1=4216000000000000 fpr2=40a8000000000000 fpr3=3fb3eb36cd964838
j> 07:05:43 fpr4=3ff0000000000000 fpr5=3f9b900000000000 fpr6=322bcc763d5fb500 fpr7=40a7f9a000000000
j> 07:05:43 fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000
j> 07:05:43 fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000
j> 07:05:43 Program_Unit_Name=
j> 07:05:43 Program_Unit_Address=1B98BA10 Entry_Name=hash_jit_artifact_array_insert
j> 07:05:43 Entry_Address=1B98BA10
j> 07:05:43 
j> 07:05:43 Method_being_compiled=sun/reflect/GeneratedMethodAccessor622.<init>()V
j> 07:05:43 Target=2_90_20240307_67118 (z/OS 02.04.00)
j> 07:05:43 CPU=s390 (2 logical CPUs) (0xfac84000 RAM)
j> 07:05:43 ----------- Stack Backtrace -----------
j> 07:05:43 protectedIntrospectBacktraceSymbols+0x84 (, 0x1A30F764)
j> 07:05:43 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 07:05:43 omrintrospect_backtrace_symbols_ex+0x26c (, 0x1A30FA24)
j> 07:05:43 generateDiagnosticFiles+0x142 (, 0x19F70112)
j> 07:05:43 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 07:05:43 structuredSignalHandler+0x33a (?0x19F71A4A)
j> 07:05:43 mainSynchSignalHandler+0x3ea (, 0x1A2F8B4A)
j> 07:05:43 __zerro+0x1014 (, 0x196B4164)
j> 07:05:43 __zerros+0x1f6 (, 0x196B310E)
j> 07:05:43 CEEVROND+0x127c (, 0x1910A6A4)
j> 07:05:43 (, 0x18FDCC80 [CEEHDSP+0xe70])
j> 07:05:43 (, 0x18FEC3E2 [CEEHRNUH+0x9a])
j> 07:05:43 hash_jit_artifact_array_insert+0x22e ( 0x1B98BC3E)
j> 07:05:43 hash_jit_artifact_insert_range+0x84 ( 0x1B98BF34)
j> 07:05:43 TR_TranslationArtifactManager::insertArtifact(J9JITExceptionTable*)+0x98 (, 0x1A8A4C60)
j> 07:05:43 createMethodMetaData(TR_J9VMBase&,TR_ResolvedMethod*,TR::Compilation*)+0xbd0 (, 0x1A910740)
j> 07:05:43 TR::CompilationInfoPerThreadBase::compile(J9VMThread*,TR::Compilation*,TR_ResolvedMethod*,T...+0xe8e (, 0x1A654E56)
j> 07:05:43 TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*,void*)+0x22bc (, 0x1A652E1C)
j> 07:05:43 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 07:05:43 TR::CompilationInfoPerThreadBase::compile(J9VMThread*,TR_MethodToBeCompiled*,J9::J9SegmentP...+0x5b4 (, 0x1A648F5C)
j> 07:05:43 TR::CompilationInfoPerThread::processEntry(TR_MethodToBeCompiled&,J9::J9SegmentProvider&)+0x726 (, 0x1A6481EE)
j> 07:05:43 TR::CompilationInfoPerThread::processEntries()+0x592 (, 0x1A646872)
j> 07:05:43 protectedCompilationThreadProc(J9PortLibrary*,TR::CompilationInfoPerThread*)+0x35c (, 0x1A646034)
j> 07:05:43 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 07:05:43 compilationThreadProc(void*)+0x396 (, 0x1A643606)
j> 07:05:43 thread_wrapper+0x8fc (, 0x1A218144)
j> 07:05:43 CEEVROND+0x127c (, 0x1910A6A4)
j> 07:05:43 (, 0x0000DC6E [CEEOPCMM+0x986])
pshipton commented 4 months ago

@hzongaro fyi

hzongaro commented 3 months ago

@dsouzai, may I ask you to take a look at this?

dsouzai commented 3 months ago

Yay, 31-bit z/OS 😭

dsouzai commented 3 months ago

Initial investigation info dump:

(kca) where
0x1b98bc44 {libj9jit29.so}{hash_jit_artifact_array_insert} [0x3d5fc7e0]
0x1b98bf38 {libj9jit29.so}{hash_jit_artifact_insert_range} [0x3d5fc860]
0x1a8a4c62 {libj9jit29.so}{TR_TranslationArtifactManager::insertArtifact(J9JITExceptionTable*)} [0x3d5fc8e0]
(kca) regs
  $r0 0x00000000ff7ffff9   $r1 0x0000000019f234f8   $r2 0x000000004c511320   $r3 0x0000000047e83e84
  $r4 0x000000003d5fbfe0   $r5 0x000000001bdea8f8   $r6 0x000000000000005c   $r7 0x00000000000a0000
  $r8 0x000000008000004f   $r9 0x0000000047e84588  $r10 0x0000000000000710  $r11 0x000000004b52ddfc
 $r12 0x00000000199894b8  $r13 0x000000004b52deb4  $r14 0x0000000000000000  $r15 0x000000004a719b48
$psw0 0x078d1400
$addr 0x9b98bc44
 $bea 0x1b98bc4c

Disassembly for hash_jit_artifact_insert_range(J9PortLibrary *portLibrary, J9JITHashTable *table, J9JITExceptionTable *dataToInsert, UDATA startPC, UDATA endPC):

(kca) x/50i 0x1b98beb0
0x1b98beb0 {libj9jit29.so}{hash_jit_artifact_insert_range} +0   905C4784     STM       GPR5,GPR12,0x784(GPR4)
0x1b98beb4 {libj9jit29.so}{hash_jit_artifact_insert_range} +4   E3404F80FF71 LAY       GPR4,-128(,GPR4)
0x1b98beba {libj9jit29.so}{hash_jit_artifact_insert_range} +10  586048CC     L         GPR6,0x8CC(,GPR4)
0x1b98bebe {libj9jit29.so}{hash_jit_artifact_insert_range} +14  580048D0     L         GPR0,0x8D0(,GPR4)
0x1b98bec2 {libj9jit29.so}{hash_jit_artifact_insert_range} +18  503048C8     ST        GPR3,0x8C8(,GPR4)
0x1b98bec6 {libj9jit29.so}{hash_jit_artifact_insert_range} +22  501048C0     ST        GPR1,0x8C0(,GPR4)
0x1b98beca {libj9jit29.so}{hash_jit_artifact_insert_range} +26  502048C4     ST        GPR2,0x8C4(,GPR4)
0x1b98bece {libj9jit29.so}{hash_jit_artifact_insert_range} +30  5560200C     CL        GPR6,0xC(,GPR2)
0x1b98bed2 {libj9jit29.so}{hash_jit_artifact_insert_range} +34  41300001     LA        GPR3,0x1
0x1b98bed6 {libj9jit29.so}{hash_jit_artifact_insert_range} +38  A744005A     JL        *0xB4 C>> +218
0x1b98beda {libj9jit29.so}{hash_jit_artifact_insert_range} +42  55002010     CL        GPR0,0x10(,GPR2)
0x1b98bede {libj9jit29.so}{hash_jit_artifact_insert_range} +46  A7240056     JH        *0xAC C>> +218
0x1b98bee2 {libj9jit29.so}{hash_jit_artifact_insert_range} +50  5F60200C     SL        GPR6,0xC(,GPR2)
0x1b98bee6 {libj9jit29.so}{hash_jit_artifact_insert_range} +54  5F00200C     SL        GPR0,0xC(,GPR2)
0x1b98beea {libj9jit29.so}{hash_jit_artifact_insert_range} +58  88600009     SRL       GPR6,0x9
0x1b98beee {libj9jit29.so}{hash_jit_artifact_insert_range} +62  88000009     SRL       GPR0,0x9
0x1b98bef2 {libj9jit29.so}{hash_jit_artifact_insert_range} +66  89600002     SLL       GPR6,0x2
0x1b98bef6 {libj9jit29.so}{hash_jit_artifact_insert_range} +70  5E602008     AL        GPR6,0x8(,GPR2)
0x1b98befa {libj9jit29.so}{hash_jit_artifact_insert_range} +74  89000002     SLL       GPR0,0x2
0x1b98befe {libj9jit29.so}{hash_jit_artifact_insert_range} +78  50604864     ST        GPR6,0x864(,GPR4)
0x1b98bf02 {libj9jit29.so}{hash_jit_artifact_insert_range} +82  5E002008     AL        GPR0,0x8(,GPR2)
0x1b98bf06 {libj9jit29.so}{hash_jit_artifact_insert_range} +86  50004868     ST        GPR0,0x868(,GPR4)
0x1b98bf0a {libj9jit29.so}{hash_jit_artifact_insert_range} +90  58306000     L         GPR3,0x0(,GPR6) <<< ^+194
0x1b98bf0e {libj9jit29.so}{hash_jit_artifact_insert_range} +94  EC380024007E CIJ       GPR3,0,JE,*0x48 C>> +166
0x1b98bf14 {libj9jit29.so}{hash_jit_artifact_insert_range} +100 580048C8     L         GPR0,0x8C8(,GPR4)
0x1b98bf18 {libj9jit29.so}{hash_jit_artifact_insert_range} +104 586048CC     L         GPR6,0x8CC(,GPR4)
0x1b98bf1c {libj9jit29.so}{hash_jit_artifact_insert_range} +108 58C04820     L         GPR12,0x820(,GPR4)
0x1b98bf20 {libj9jit29.so}{hash_jit_artifact_insert_range} +112 581048C0     L         GPR1,0x8C0(,GPR4)
0x1b98bf24 {libj9jit29.so}{hash_jit_artifact_insert_range} +116 582048C4     L         GPR2,0x8C4(,GPR4)
0x1b98bf28 {libj9jit29.so}{hash_jit_artifact_insert_range} +120 58504804     L         GPR5,0x804(,GPR4)
0x1b98bf2c {libj9jit29.so}{hash_jit_artifact_insert_range} +124 50604850     ST        GPR6,0x850(,GPR4)
0x1b98bf30 {libj9jit29.so}{hash_jit_artifact_insert_range} +128 5000484C     ST        GPR0,0x84C(,GPR4)
0x1b98bf34 {libj9jit29.so}{hash_jit_artifact_insert_range} +132 A775FD6E     JAS       GPR7,*-0x524 (0x1B98BA10) ^{libj9jit29.so}{hash_jit_artifact_array_insert} +0
...

Partial disassembly of hash_jit_artifact_array_insert(J9PortLibrary *portLibrary, J9JITHashTable *table, J9JITExceptionTable** array, J9JITExceptionTable *dataToInsert, UDATA startPC):

0x1b98ba10 {libj9jit29.so}{hash_jit_artifact_array_insert} +0   906A4788     STM       GPR6,GPR10,0x788(GPR4)
0x1b98ba14 {libj9jit29.so}{hash_jit_artifact_array_insert} +4   E3404F80FF71 LAY       GPR4,-128(,GPR4)
0x1b98ba1a {libj9jit29.so}{hash_jit_artifact_array_insert} +10  502048C4     ST        GPR2,0x8C4(,GPR4)
0x1b98ba1e {libj9jit29.so}{hash_jit_artifact_array_insert} +14  1892         LR        GPR9,GPR2
0x1b98ba20 {libj9jit29.so}{hash_jit_artifact_array_insert} +16  A7310001     TMLL      GPR3,1
0x1b98ba24 {libj9jit29.so}{hash_jit_artifact_array_insert} +20  18A3         LR        GPR10,GPR3
0x1b98ba26 {libj9jit29.so}{hash_jit_artifact_array_insert} +22  50304864     ST        GPR3,0x864(,GPR4)
0x1b98ba2a {libj9jit29.so}{hash_jit_artifact_array_insert} +26  A7840054     JE        *0xA8 C>> +194
0x1b98ba2e {libj9jit29.so}{hash_jit_artifact_array_insert} +30  58302020     L         GPR3,0x20(,GPR2)
0x1b98ba32 {libj9jit29.so}{hash_jit_artifact_array_insert} +34  41003008     LA        GPR0,0x8(,GPR3)
0x1b98ba36 {libj9jit29.so}{hash_jit_artifact_array_insert} +38  5900201C     C         GPR0,0x1C(,GPR2)
0x1b98ba3a {libj9jit29.so}{hash_jit_artifact_array_insert} +42  A7D4003E     JNH       *0x7C C>> +166
0x1b98ba3e {libj9jit29.so}{hash_jit_artifact_array_insert} +46  587011DC     L         GPR7,0x1DC(,GPR1)
0x1b98ba42 {libj9jit29.so}{hash_jit_artifact_array_insert} +50  C0098000000B IILF      GPR0,0x8000000B
0x1b98ba48 {libj9jit29.so}{hash_jit_artifact_array_insert} +56  41200408     LA        GPR2,0x408
0x1b98ba4c {libj9jit29.so}{hash_jit_artifact_array_insert} +60  C03000000A56 LARL      GPR3,*0x14AC
0x1b98ba52 {libj9jit29.so}{hash_jit_artifact_array_insert} +66  5000484C     ST        GPR0,0x84C(,GPR4)
0x1b98ba56 {libj9jit29.so}{hash_jit_artifact_array_insert} +70  58607014     L         GPR6,0x14(,GPR7)
0x1b98ba5a {libj9jit29.so}{hash_jit_artifact_array_insert} +74  58507010     L         GPR5,0x10(,GPR7)
0x1b98ba5e {libj9jit29.so}{hash_jit_artifact_array_insert} +78  0D76         BASR      GPR7,GPR6
0x1b98ba60 {libj9jit29.so}{hash_jit_artifact_array_insert} +80  47000041     NOP       0x41
0x1b98ba64 {libj9jit29.so}{hash_jit_artifact_array_insert} +84  EC3800D3007E CIJ       GPR3,0,JE,*0x1A6 C>> +506
...

Get the stack pointer in the hash_jit_artifact_insert_range frame:

0x000000003d5fbfe0+128 = 0x3D5FC060

Get the J9JITExceptionTable (save of GPR3):

(kca) what (0x3D5FC060+0x84C)/a
0x3d5fc8ac: 0x4a719b48 Ptr Unknown!
(kca) struct J9JITExceptionTable 0x4a719b48
J9JITExceptionTable (116 bytes)
                           struct J9UTF8 *  className = 0x47f4d528 (offset: 0)
                          struct J9UTF8 *  methodName = 0x3e752498 (offset: 4)
                     struct J9UTF8 *  methodSignature = 0x3e7524a0 (offset: 8)
                struct J9ConstantPool *  constantPool = 0x480be6e0 (offset: 12)
                         struct J9Method *  ramMethod = 0x480be764 (offset: 16)
                                       UDATA  startPC = 0x4b52ddfc (offset: 20)
                                     UDATA  endWarmPC = 0x4b52deb4 (offset: 24)
                                   UDATA  startColdPC = 0x00000000 (offset: 28)
                                         UDATA  endPC = 0x4b52deb4 (offset: 32)
                                UDATA  totalFrameSize = 0x00000007 (offset: 36)
                                          I_16  slots = 0x00000001 (offset: 40)
                                I_16  scalarTempSlots = 0x00000005 (offset: 42)
                                I_16  objectTempSlots = 0x00000000 (offset: 44)
                                 U_16  prologuePushes = 0x00000000 (offset: 46)
                                     I_16  tempOffset = 0x00000000 (offset: 48)
                              U_16  numExcptionRanges = 0x00000000 (offset: 50)
                                           I_32  size = 0x000000d2 (offset: 52)
                                         UDATA  flags = 0x00000000 (offset: 56)
                       UDATA  registerSaveDescription = 0x00180000 (offset: 60)
                                 void *  gcStackAtlas = 0x4a719bc5 (offset: 64)
                                 void *  inlinedCalls = 0x4a719bbc (offset: 68)
                                     void *  bodyInfo = 0x4adb9138 (offset: 72)
             struct J9JITExceptionTable *  nextMethod = 0x00000000 (offset: 76)
             struct J9JITExceptionTable *  prevMethod = 0x00000000 (offset: 80)
                                   void *  debugSlot1 = 0x00000000 (offset: 84)
                                   void *  debugSlot2 = 0x00000000 (offset: 88)
                                      void *  osrInfo = 0x00000000 (offset: 92)
                        void *  runtimeAssumptionList = 0x00000000 (offset: 96)
                                        I_32  hotness = 0x00000002 (offset: 100)
                                UDATA  codeCacheAlloc = 0x4b52dccc (offset: 104)
                                      void *  gpuCode = 0x00000000 (offset: 108)
                                       void *  riData = 0x00000000 (offset: 112)
                     JIT_METADATA_FLAGS_USED_FOR_SIZE = 0x0 / 0x1 (constant)
                   JIT_METADATA_GC_MAP_32_BIT_OFFSETS = 0x0 / 0x2 (constant)
                    JIT_METADATA_IS_DESERIALIZED_COMP = 0x0 / 0x20 (constant)
                             JIT_METADATA_IS_FSD_COMP = 0x0 / 0x80 (constant)
                   JIT_METADATA_IS_PRECHECKPOINT_COMP = 0x0 / 0x40 (constant)
                          JIT_METADATA_IS_REMOTE_COMP = 0x0 / 0x10 (constant)
                                 JIT_METADATA_IS_STUB = 0x0 / 0x4 (constant)
                         JIT_METADATA_NOT_INITIALIZED = 0x0 / 0x8 (constant)
(kca) j9m 0x480be764
Method   {ClassPath/Name.MethodName}: {sun/reflect/GeneratedMethodAccessor622.<init>}
                           Signature: ()V
                              Access: Public
                    J9Class/J9Method: 0x480be600 / 0x480be764
               Compiled Method Start: Not Compiled! (count=0)
                      ByteCode Start: 0x47f4d484 (5 bytes)
                   ROM Constant Pool: 0x47f4d3c8 (16 entries)
                       Constant Pool: 0x480be6e0 (0 entries)

From hash_jit_artifact_array_insert:

0x1b98ba20 {libj9jit29.so}{hash_jit_artifact_array_insert} +16  A7310001     TMLL      GPR3,1
0x1b98ba24 {libj9jit29.so}{hash_jit_artifact_array_insert} +20  18A3         LR        GPR10,GPR3

==> GPR10 is J9JITExceptionTable** array

Seems like the J9JITExceptionTable** array passed to hash_jit_artifact_array_insert is garbage (0x00000710). The relevant code is: https://github.com/eclipse-openj9/openj9/blob/e77cdfc1aec3f78bd0ae6c2d0b9cf3909828aeb8/runtime/codert_vm/jithash.cpp#L179-L199

Get the startPC (sanity check: it matches the startPC from the J9JITExceptionTable to be inserted):

(kca) what (0x3D5FC060+0x8CC)/a
0x3d5fc92c: 0x4b52ddfc Ptr Unknown!

It seems to be a valid for the range specified by the J9JITHashTable:

(kca) struct J9JITHashTable 0x4c511320
J9JITHashTable (36 bytes)
                     J9AVLTreeNode  parentAVLTreeNode = 0x0000000000000000 (offset: 0)
                                 uintptr_t *  buckets = 0x4a2ccbd0 (offset: 8)
                                     uintptr_t  start = 0x4b400000 (offset: 12)
                                       uintptr_t  end = 0x4b600000 (offset: 16)
                                     uintptr_t  flags = 0x00000000 (offset: 20)
                        uintptr_t *  methodStoreStart = 0x47e83c20 (offset: 24)
                          uintptr_t *  methodStoreEnd = 0x47e84024 (offset: 28)
                         uintptr_t *  currentAllocate = 0x47e83ed8 (offset: 32)
                               JIT_HASH_IN_DATA_CACHE = 0x0 / 0x1 (constant)
dsouzai commented 3 months ago

I'll get back to this another day; the next step is going to be the very tedious iteration of

 do { 
    if (*index) { 
        temp = hash_jit_artifact_array_insert(portLibrary, table, (J9JITExceptionTable**) *index, dataToInsert, startPC); 
        if (!temp) { 
            return 2; 
        } 
        VM_AtomicSupport::writeBarrier(); 
        *index = (J9JITExceptionTable *) temp; 
    } else { 
        VM_AtomicSupport::writeBarrier(); 
        *index = (J9JITExceptionTable *) SET_LOW_BIT(dataToInsert); 
    } 

 } while (++index <= endIndex);

to see why J9JITExceptionTable** array ended up as 0x00000710.

pshipton commented 2 weeks ago

Since reproduction is rare (only seen once) and we don't have a solution, I've removed the milestone target.