search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

abbs J9::CompilationStrategy::ProcessJittedSample::process() crash vmState=0x00000000 #18953

Open pshipton opened 7 months ago

pshipton commented 7 months ago

Internal build [zOS S390] 80 Load_Level_2.abbs.5mins.Mode121 -Xgcpolicy:optavgpause -Xjit:count=0 -Xnocompressedrefs - fyrec607

130x grinder - passed

j> 13:59:57 20240213 13:59:56 Runtime State Reporter IMPORTANT: 9178 tests complete, 275 currently running
j> 14:00:08 Unhandled exception
j> 14:00:08 Type=Segmentation error vmState=0x00000000
j> 14:00:08 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000035
j> 14:00:08 Handler1=1995A498 Handler2=1A35C098
j> 14:00:08 gpr0=10200000 gpr1=3D0657AC gpr2=00000020 gpr3=00000000
j> 14:00:08 gpr4=41F5D220 gpr5=1A2358B8 gpr6=000A0000 gpr7=9A68A9BA
j> 14:00:08 gpr8=1BE1F760 gpr9=41F5DBC8 gpr10=00000000 gpr11=1BE98E5C
j> 14:00:08 gpr12=41DBD708 gpr13=1A68B770 gpr14=41F5DE86 gpr15=3D9D3DB0
j> 14:00:08 hgpr0=0000F71F hgpr1=00000000 hgpr2=00000000 hgpr3=00000000
j> 14:00:08 hgpr4=000063F2 hgpr5=00000000 hgpr6=00000000 hgpr7=00000000
j> 14:00:08 hgpr8=000063F2 hgpr9=0000FFFF hgpr10=00000005 hgpr11=00000000
j> 14:00:08 hgpr12=00000000 hgpr13=00000000 hgpr14=00000000 hgpr15=00000000
j> 14:00:08 fpc=0008fe00 psw0=078D3400 psw1=9A68A9CA sp=41F5D220
j> 14:00:08 bea=1A824864
j> 14:00:08 fpr0=0000000000000000 fpr1=3e206435816bc5ca fpr2=0000000500000000 fpr3=bfc5af496df1307d
j> 14:00:08 fpr4=0000000000000000 fpr5=40a5bd2e147ae148 fpr6=3f8f012875c5d5b5 fpr7=40a5bbae147ae148
j> 14:00:08 fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000
j> 14:00:08 fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000
j> 14:00:08 Program_Unit_Name=
j> 14:00:08 Program_Unit_Address=1A68A6F0 Entry_Name=J9::CompilationStrategy::ProcessJittedSample::process()
j> 14:00:08 Entry_Address=1A68A6F0
j> 14:00:08 vr0=00000000000000000000000000000000 vr1=3e206435816bc5ca0000000000000000 vr2=00000005000000000000000000000000 vr3=bfc5af496df1307d0000000000000000
j> 14:00:08 vr4=00000000000000000000000000000000 vr5=40a5bd2e147ae1480000000000000000 vr6=3f8f012875c5d5b50000000000000000 vr7=40a5bbae147ae1480000000000000000
j> 14:00:08 vr8=00000000000000000000000000000000 vr9=00000000000000000000000000000000 vr10=00000000000000000000000000000000 vr11=00000000000000000000000000000000
j> 14:00:08 vr12=00000000000000000000000000000000 vr13=00000000000000000000000000000000 vr14=00000000000000000000000000000000 vr15=00000000000000000000000000000000
j> 14:00:08 vr16=00650073007400450044003200500044 vr17=00000000000000100000000000000000 vr18=002e002e002e002e002e002e002e002e vr19=002000640075006d0070006f006e0066
j> 14:00:08 vr20=00000000000000000000000000000000 vr21=00000000000000000000000000000000 vr22=00000000000000000000000000000000 vr23=00000000000000000000000000000000
j> 14:00:08 vr24=00000000000000000000000000000000 vr25=00000000000000000000000000000000 vr26=00000000000000000000000000000000 vr27=00000000000000000000000000000000
j> 14:00:08 vr28=00000000000000000000000000000000 vr29=00000000000000000000000000000000 vr30=00000000000000000000000000000000 vr31=00000000000000000000000000000000
j> 14:00:08 Target=2_90_20240213_65722 (z/OS 02.04.00)
j> 14:00:08 CPU=s390 (2 logical CPUs) (0xfac53000 RAM)
j> 14:00:08 ----------- Stack Backtrace -----------
j> 14:00:08 protectedIntrospectBacktraceSymbols+0x84 (, 0x1A30F764)
j> 14:00:08 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 14:00:08 omrintrospect_backtrace_symbols_ex+0x26c (, 0x1A30FA24)
j> 14:00:08 generateDiagnosticFiles+0x142 (, 0x19F70112)
j> 14:00:08 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 14:00:08 structuredSignalHandler+0x33a (?0x19F71A4A)
j> 14:00:08 mainSynchSignalHandler+0x3ea (, 0x1A2F8B4A)
j> 14:00:08 __zerro+0x1014 (, 0x196B3474)
j> 14:00:08 __zerros+0x1f6 (, 0x196B241E)
j> 14:00:08 CEEVROND+0x127c (, 0x1910A6A4)
j> 14:00:08 (, 0x18FDCC80 [CEEHDSP+0xe70])
j> 14:00:08 (, 0x18FEC3E2 [CEEHRNUH+0x9a])
j> 14:00:08 J9::CompilationStrategy::ProcessJittedSample::process()+0x2da (, 0x1A68A9CA)
j> 14:00:08 J9::CompilationStrategy::processEvent(TR_MethodEvent*,bool*)+0x792 (, 0x1A687AEA)
j> 14:00:08 J9::Recompilation::sampleMethod(void*,TR_FrontEnd*,void*,int,void*,void*,int)+0x132 (, 0x1A907A62)
j> 14:00:08 jitMethodSampleInterrupt(J9VMThread*,long,void*)+0x8be (, 0x1A676BCE)
j> 14:00:08 dispatchAsyncEvents+0x166 ( 0x19F7569E)
j> 14:00:08 javaCheckAsyncMessages+0xae (0x1A000ED6)
j> 14:00:08 old_slow_jitStackOverflow+0x2a2 (, 0x1B99B4C2)
j> 14:00:08 ZJ9SYM1+0x5a733e (, 0x1A4D2626)
j> 14:00:08 sidecarInvokeReflectMethodImpl+0x930 (, 0x19F3F728)
j> 14:00:08 sidecarInvokeReflectMethod+0x80 (, 0x19F40E80)
j> 14:00:08 JVM_InvokeMethod_Impl+0x78 ( 0x1C7B3C90)
j> 14:00:08 JVM_InvokeMethod+0x100 (??, 0x19EE58A0)
j> 14:00:08 Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x1c (, 0x1A26FE1C)
j> 14:00:08 ZJ9SYM1+0x251ac200 (, 0x3F0D74E8)
j> 14:00:08 runJavaThread+0x4be ( 0x19F37FBE)
j> 14:00:08 javaProtectedThreadProc(J9PortLibrary*,void*)+0xe8 (, 0x19FF8EE0)
j> 14:00:08 omrsig_protect+0xa98 (, 0x1A2FDCF0)
j> 14:00:08 javaThreadProc+0x60 ( 0x19FF8D70)
j> 14:00:08 thread_wrapper+0x8fc (, 0x1A218144)
j> 14:00:08 CEEVROND+0x127c (, 0x1910A6A4)
j> 14:00:08 (, 0x0000DC6E [CEEOPCMM+0x986])
j> 14:00:08 ---------------------------------------
pshipton commented 7 months ago

@hzongaro fyi

0xdaryl commented 7 months ago

@mpirvu : can you assign for investigation please?

dsouzai commented 6 months ago

@mpirvu I can take a look at this since I was the one to refactor all that code.

dsouzai commented 6 months ago

Registers:

gpr0=10200000 gpr1=3D0657AC gpr2=00000020 gpr3=00000000
gpr4=41F5D220 gpr5=1A2358B8 gpr6=000A0000 gpr7=9A68A9BA
gpr8=1BE1F760 gpr9=41F5DBC8 gpr10=00000000 gpr11=1BE98E5C
gpr12=41DBD708 gpr13=1A68B770 gpr14=41F5DE86 gpr15=3D9D3DB0
hgpr0=0000F71F hgpr1=00000000 hgpr2=00000000 hgpr3=00000000
hgpr4=000063F2 hgpr5=00000000 hgpr6=00000000 hgpr7=00000000
hgpr8=000063F2 hgpr9=0000FFFF hgpr10=00000005 hgpr11=00000000
hgpr12=00000000 hgpr13=00000000 hgpr14=00000000 hgpr15=00000000
fpc=0008fe00 psw0=078D3400 psw1=9A68A9CA sp=41F5D220
bea=1A824864

The crash happens at 0x1a68a9ca

0x1a68a9c2 {libj9jit29.so}{J9::CompilationStrategy::ProcessJittedSample::process()} +722 58609024     L         GPR6,0x24(,GPR9)
0x1a68a9c6 {libj9jit29.so}{J9::CompilationStrategy::ProcessJittedSample::process()} +726 58606000     L         GPR6,0x0(,GPR6)
0x1a68a9ca {libj9jit29.so}{J9::CompilationStrategy::ProcessJittedSample::process()} +730 5800600C     L         GPR0,0xC(,GPR6)

Essentially, gpr9 is the this pointer of the J9::CompilationStrategy::ProcessJittedSample class, and GPR6,0x24(,GPR9) is doing this->_methodInfo which is NULL:

> x/50x 0x41F5DBC8

        0x41f5dbc8: 19adf520 // _jitConfig        // +0x0
        0x41f5dbcc: 418ea100 // _vmThread         // +0x4
        0x41f5dbd0: 19ae0720 // _compInfo         // +0x8
        0x41f5dbd4: 40e63e20 // _fe               // +0xC
        0x41f5dbd8: 3d2064b8 // _cmdLineOptions   // +0x10
        0x41f5dbdc: 3d9d3db0 // _j9method         // +0x14
        0x41f5dbe0: 41f5dea8 // _event            // +0x18
        0x41f5dbe4: 3f0a5af8 // _startPC          // +0x1C
        0x41f5dbe8: 3eb77408 // _bodyInfo         // +0x20
        0x41f5dbec: 00000000 // _methodInfo       // +0x24
        0x41f5dbf0: 00000020
        0x41f5dbf4: 418ea7bc
        0x41f5dbf8: 1a2105a8
        0x41f5dbfc: 1a4a4920
        0x41f5dc00: 41dbd708
        0x41f5dc04: 1a4a6f2c
        0x41f5dc08: 00000000
        0x41f5dc0c: 19fe949c
        0x41f5dc10: 19b46d20
        0x41f5dc14: 00000001

The method in question is

(kca) m 0x3f0a5af8
         Method Signature: {java/lang/StringBuilder.append(I)Ljava/lang/StringBuilder;}
                 MetaData: 0x3d7fa188 (optLevel: warm)
               Frame Size: 60 bytes
                   Access: Public
         J9Class/J9Method: 0x3d9d4800 / 0x3d9d3db0
               MethodInfo: 0x00000000
                 BodyInfo: 0x3eb77408
Compiled Method Start/End: 0x3f0a5a24 / 0x3f0a5cfc (728 bytes)
      Cold Code Start/End: 0x00000000 / 0x00000000 (0 bytes)

Method   {ClassPath/Name.MethodName}: {java/lang/StringBuilder.append}
                           Signature: (I)Ljava/lang/StringBuilder;
                              Access: Public
                    J9Class/J9Method: 0x3d9d4800 / 0x3d9d3db0
               Compiled Method Start: 0x3f0a5af8 (728 bytes)
                      ByteCode Start: 0x3d9e0abc (139 bytes)
                   ROM Constant Pool: 0x3d9dfc50 (144 entries)
                       Constant Pool: 0x3d9d4180 (1 entries)

The first few fields of the TR_PersistentJittedBodyInfo is 

   private:
   int32_t                  _counter;             // must be at offset 0
   TR_PersistentMethodInfo *_methodInfo;          // must be at offset 4 (8 for 64bit)
   void                    *_startPCAfterPreviousCompile;
   void                    *_mapTable;            // must be at offset 12 (24 for 64bit)

which maps to

(kca) x/4x 0x3eb77408
0x3eb77408: 1cb3a398 00000000 00000000 1cd73028

The body info pointer is the same in the exception table:

(kca) struct J9JITExceptionTable.bodyInfo 0x3d7fa188
J9JITExceptionTable (116 bytes)
                                     void *  bodyInfo = 0x3eb77408 (offset: 72)

as well as in the code

(kca) x/x 0x3f0a5af8-8
0x3f0a5af0: 3eb77408

However, from all indication, the bodyInfo looks corrupted. I also can't find the methodInfo anywhere in the Internal Memory segments, so it leads me to believe that this memory got overwritten somehow.

The JVM wasn't using AOT so there wasn't any relocation issue, and as far as I can tell, there's no redefinition happening. Also class unloading couldn't be a factor as java/lang/StringBuilder.append is on the stack.

I don't really have any theories as to what is going on except for may be memory corruption or an erroneous free.