search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.29k stars 723 forks source link

z/OS abbs memoryCorruptionDetected from jitReleaseCodeCollectMetaData #14085

Open pshipton opened 2 years ago

pshipton commented 2 years ago

/job_output.php?id=21841214 [zOS S390 64bit] 80 Load_Level_2.abbs.5mins.Mode103.5 - fyrec604 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs

80x grinder passed

j> 19:36:01 20211205 19:35:58 Runtime State Reporter IMPORTANT: 2816 tests complete, 300 currently running
j> 19:36:23 03:36:22.514 0x5102581a00 omrport.359    *   ** ASSERTION FAILED ** at ./../../omr/port/common/omrmemtag.c:145: ((memoryCorruptionDetected))
0x0: void * corruptedMemoryBlock = !j9x 0x0000005C2328CA30
> !j9x 0x0000005C2328C900,200
0x5C2328C900 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C910 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C920 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C930 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C940 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C950 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C960 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C970 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C980 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C990 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C9A0 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C9B0 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C9C0 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C9D0 :  0000000000000000 0000000000000000 [ ................ ] 
0x5C2328C9E0 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328C9F0 :  63c32d0000000000 0000000000000000 [ c.-............. ] 
0x5C2328CA00 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] 
0x5C2328CA10 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328CA20 :  63c32d0000000000 0000000000000000 [ c.-............. ] 
0x5C2328CA30 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] 
0x5C2328CA40 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328CA50 :  63c32d0000000000 0000000000000000 [ c.-............. ] 
0x5C2328CA60 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] 
0x5C2328CA70 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328CA80 :  63c32d0000000000 0000000000000000 [ c.-............. ] 
0x5C2328CA90 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] 
0x5C2328CAA0 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328CAB0 :  63c32d0000000000 0000000000000000 [ c.-............. ] 
0x5C2328CAC0 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] 
0x5C2328CAD0 :  0000000000000000 0000000000000050 [ ...............P ] 
0x5C2328CAE0 :  63c32d0000000000 000000000000023e [ c.-............> ] 
0x5C2328CAF0 :  0000000000000070 0000000700000000 [ .......p........ ] 
info thread
 process id: 659

  thread id: 0x19776000
   registers:
    PSW    = 0x078d140099bbb082   R0     = 0x0000000000000000   R1     = 0x0000000019c7c3e0   R2     = 0x0000000019c7b408
    R3     = 0x0000000019c7c3e0   R4     = 0x000000524c5f9060   R5     = 0x0000000000000000   R6     = 0x0000000019bbafc0
    R7     = 0x0000000019c10da2   R8     = 0x0000000019bbafd2   R9     = 0x000000524c5fb530   R10    = 0x0000005008852ea0
    R11    = 0x0000000019c11730   R12    = 0x0000005c38aba310   R13    = 0x0000005c38aba310   R14    = 0x0000000019bbb124
    R15    = 0x000000007f567f00
   native stack sections:
    0x524c5f4000 to 0x524c600000 (length 0xc000)
   native stack frames:
    bp: 0x000000524c5f9060 pc: 0x0000000019bbb082 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9prt29.so::_TDUMP+0xc2
    bp: 0x000000524c5f9100 pc: 0x0000000019c10da0 ExtraSymbolsModule::tdump_wrapper+0x5a0
    bp: 0x000000524c5f9b80 pc: 0x0000000019c0ff6e ExtraSymbolsModule::omrdump_create+0x56
    bp: 0x000000524c5fa280 pc: 0x0000000019c8e686 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9dmp29.so::doSystemDump+0x38e
    bp: 0x000000524c5fa380 pc: 0x0000000019c8f32c /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9dmp29.so::protectedDumpFunction+0x34
    bp: 0x000000524c5fa480 pc: 0x0000000019c19c10 ExtraSymbolsModule::omrsig_protect+0x458
    bp: 0x000000524c5fac00 pc: 0x000000001c7fd058 libzip.so::Java_java_util_zip_ZipFile_initIDs+0x913cb8
    bp: 0x000000524c5fb180 pc: 0x0000000019cd3c22 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9dmp29.so::triggerDumpAgents+0x622
    bp: 0x000000524c5fb540 pc: 0x0000000019d23448 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9trc29.so::doTriggerActionAssertDumpEvent+0x48
    bp: 0x000000524c5fb640 pc: 0x0000000019d51d40 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9trc29.so::triggerHit+0x4f0
    bp: 0x000000524c5fb780 pc: 0x0000000019d2dfaa /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9trc29.so::logTracePoint+0x13a2
    bp: 0x000000524c5fba40 pc: 0x0000000019d36238 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9trc29.so::doTracePoint+0x230
    bp: 0x000000524c5fbb40 pc: 0x0000000019d22dfc /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9trc29.so::javaTrace+0xac
    bp: 0x000000524c5fbc40 pc: 0x0000000019bfd656 ExtraSymbolsModule::omrmem_free_memory+0x1de
    bp: 0x000000524c5fbd40 pc: 0x000000001a0f8226 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9jit29.so::jitReleaseCodeCollectMetaData__FP11J9JITConfigP10J9VMThreadP19J9JITExceptionTablePQ2_3OMR15FaintCacheBlock+0x6be
    bp: 0x000000524c5fbf00 pc: 0x0000000019e96520 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9jit29.so::jitHookReleaseCodeGlobalGCEnd(J9HookInterface**,unsigned long,void*,void*)+0x260
    bp: 0x000000524c5fc240 pc: 0x0000000019c83914 ExtraSymbolsModule::J9HookDispatch(J9HookInterface**,unsigned long,void*)+0x29c
    bp: 0x000000524c5fc3c0 pc: 0x000000001b92c65c ExtraSymbolsModule::MM_ParallelGlobalGC::internalPostCollect(MM_EnvironmentBase*,MM_MemorySubSpace*)+0xadc
    bp: 0x000000524c5fc740 pc: 0x000000001b880122 ExtraSymbolsModule::MM_Collector::postCollect(MM_EnvironmentBase*,MM_MemorySubSpace*)+0x9a
    bp: 0x000000524c5fc980 pc: 0x000000001b881670 ExtraSymbolsModule::MM_Collector::garbageCollect(MM_EnvironmentBase*,MM_MemorySubSpace*,MM_AllocateDescription*...+0xa50
    bp: 0x000000524c5fca80 pc: 0x000000001b86b94a ExtraSymbolsModule::MM_MemorySubSpace::systemGarbageCollect(MM_EnvironmentBase*,unsigned int)+0x732
    bp: 0x000000524c5fcdc0 pc: 0x000000001b86b260 ExtraSymbolsModule::MM_MemorySubSpace::systemGarbageCollect(MM_EnvironmentBase*,unsigned int)+0x48
    bp: 0x000000524c5fd100 pc: 0x000000001b8b710a ExtraSymbolsModule::MM_MemorySpace::systemGarbageCollect(MM_EnvironmentBase*,unsigned int)+0x32
    bp: 0x000000524c5fd200 pc: 0x000000001b83e266 ExtraSymbolsModule::MM_Heap::systemGarbageCollect(MM_EnvironmentBase*,unsigned int)+0x26
    bp: 0x000000524c5fd300 pc: 0x000000001b5feea2 ExtraSymbolsModule::j9gc_modron_global_collect+0x62
    bp: 0x000000524c5fd400 pc: 0x000000001be593f8 ExtraSymbolsModule::JVM_GC_Impl+0xc8
    bp: 0x000000524c5fd500 pc: 0x000000001984c228 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libjvm.so::JVM_GC+0x130
    bp: 0x000000524c5fd600 pc: 0x0000000019b8b378 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/libjava.so::Java_java_lang_Runtime_gc+0x10
    bp: 0x000000524c5fd700 pc: 0x0000000000000000 <unknown>
    bp: 0x000000524c5fdc80 pc: 0x00000000198a4070 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9vm29.so::sidecarInvokeReflectMethodImpl+0x980
    bp: 0x000000524c5fde00 pc: 0x00000000198a572a /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9vm29.so::sidecarInvokeReflectMethod+0xaa
    bp: 0x000000524c5fdf00 pc: 0x000000001be59cc0 ExtraSymbolsModule::JVM_InvokeMethod_Impl+0xb0
    bp: 0x000000524c5fe000 pc: 0x000000001984c924 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libjvm.so::JVM_InvokeMethod+0x154
    bp: 0x000000524c5fe100 pc: 0x0000000019b888a2 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/libjava.so::Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x2a
    bp: 0x000000524c5fe200 pc: 0x0000000000000000 <unknown>
    bp: 0x000000524c5fe780 pc: 0x000000001989d39c /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9vm29.so::runJavaThread+0x3a4
    bp: 0x000000524c5fe900 pc: 0x000000001994aeba /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9vm29.so::javaProtectedThreadProc+0x18a
    bp: 0x000000524c5fea00 pc: 0x0000000019c19c10 ExtraSymbolsModule::omrsig_protect+0x458
    bp: 0x000000524c5ff180 pc: 0x000000001994e8e4 /j9vm/ebcdic/builds/bld_17749/sdk/mz6480/jre/lib/s390x/default/libj9vm29.so::javaThreadProc+0x8c
    bp: 0x000000524c5ff280 pc: 0x0000000019b32a22 libj9thr29.so::thread_wrapper+0x9ba
    bp: 0x000000524c5ff380 pc: 0x00000000191154a0 ExtraSymbolsModule::CELQPCMM+0xf80
   properties:
    CAA=0x524c7fec60           CAA CEL level=0x1d
    EDB=0x50001055e8           IFA Enabled=no
    PSW:31=0x70c100080ed6b48   PTHREADID=0x1977600000000042
    Stack direction=up         TCB=0x6b0e88
    Task Completion Code=0x0   WEBFLAG2=0x00
   associated Java thread: 
    name:          Pooled Thread #52 running com.ibm.jtc.svt.lambda.TestLambdaCapture
    Thread object: com/ibm/jtc/test/jltf/threads/PooledLangThreadFactory$JavaLangPooledThread @ 0x501986fe40
    Native info:   !j9vmthread 0x5102581a00  !stack 0x5102581a00
    Daemon:        false
    ID:            64 (0x40)
    Priority:      5
    Thread.State:  RUNNABLE 
    JVMTI state:   ALIVE RUNNABLE 
    Java stack frames: 
     bp: 0x0000005b64cbeca0  method: void java/lang/Runtime.gc()  (Native Method)
      objects: 0x501940d528
     bp: 0x0000005b64cbecb8  method: void java/lang/System.gc()  source: System.java:388
      objects: <no objects in this frame>
     bp: 0x0000005b64cbed40  method: void com/ibm/jtc/svt/lambda/TestLambdaCapture.testCaptureReturnedLambda()  source: TestLambdaCapture.java:153
      objects: 0x502707b578 0x501a5cb5e8 0x502707b578 0x50270747d0 0x502706da28
     bp: 0x0000005b64cbeb60  method: Object sun/reflect/NativeMethodAccessorImpl.invoke0(reflect.Method, Object, Object[])  (Native Method)
      objects: <no objects in this frame>
     bp: 0x0000005b64cbee78  method: Object sun/reflect/NativeMethodAccessorImpl.invoke(Object, Object[])  source: NativeMethodAccessorImpl.java:90
      objects: 0x5027066c48 0x50270052d8 0x5027066c60 0x5027066bb0
     bp: 0x0000005b64cbeea8  method: Object sun/reflect/DelegatingMethodAccessorImpl.invoke(Object, Object[])  source: DelegatingMethodAccessorImpl.java:55
      objects: <no objects in this frame>
     bp: 0x0000005b64cbef40  method: Object java/lang/reflect/Method.invoke(Object, Object[])  source: Method.java:508
      objects: <no objects in this frame>
     bp: 0x0000005b64cbefb8  method: void junit/framework/TestCase.runTest()  source: TestCase.java:168
      objects: 0x50270052d8 0x5027066bb0
     bp: 0x0000005b64cbf010  method: void junit/framework/TestCase.runBare()  source: TestCase.java:134
      objects: 0x50270052d8
     bp: 0x0000005b64cbf030  method: void junit/framework/TestResult$1.protect()  source: TestResult.java:110
      objects: <no objects in this frame>
     bp: 0x0000005b64cbf078  method: void junit/framework/TestResult.runProtected(junit.framework.Test, junit.framework.Protectable)  source: TestResult.java:128
      objects: 0x50270052d8 0x5027066798
     bp: 0x0000005b64cbf0c8  method: void junit/framework/TestResult.run(junit.framework.TestCase)  source: TestResult.java:113
      objects: 0x50270052d8 0x5027066798
     bp: 0x0000005b64cbf0f0  method: void junit/framework/TestCase.run(junit.framework.TestResult)  source: TestCase.java:124
      objects: <no objects in this frame>
     bp: 0x0000005b64cbf130  method: void junit/framework/TestSuite.runTest(junit.framework.Test, junit.framework.TestResult)  source: TestSuite.java:243
      objects: 0x50270052f0
     bp: 0x0000005b64cbf1b8  method: void junit/framework/TestSuite.run(junit.framework.TestResult)  source: TestSuite.java:238
      objects: 0x5027066798 0x50270052f0 0x5027066798 0x5027066a00 0x5027066798 0x50270052f0
     bp: 0x0000005b64cbf228  method: com.ibm.jtc.test.jltf.WorkResult com/ibm/jtc/test/jltf/workunitadapters/JUnitWorkUnit.runWork()  source: JUnitWorkUnit.java:370
      objects: 0x5027004a30 0x5027066828 0x5027066688 0x5027066688 0x5027066828
     bp: 0x0000005b64cbf2c8  method: void com/ibm/jtc/test/jltf/threads/WorkUnitRunner.run()  source: WorkUnitRunner.java:113
      objects: 0x5027066338 0x5027066338
     bp: 0x0000005b64cbf340  method: void com/ibm/jtc/test/jltf/threads/pooling/PooledRunnable.runWork()  source: PooledRunnable.java:170
      objects: 0x501986ff58 0x501986ff58
     bp: 0x0000005b64cbf3c0  method: void com/ibm/jtc/test/jltf/threads/pooling/PooledRunnable.getAndRunWork()  source: PooledRunnable.java:136
      objects: 0x501986ff58 0x501986ff58 0x501986ff58 0x501986ff58
     bp: 0x0000000000000000  method: void com/ibm/jtc/test/jltf/threads/pooling/PooledRunnable.run()  source: PooledRunnable.java:93
      objects: <no objects in this frame>
     bp: 0x0000005b64cbf420  method: void com/ibm/jtc/test/jltf/threads/PooledLangThreadFactory$JavaLangPooledThread.run()  source: PooledLangThreadFactory.java:152
      objects: 0x501986ff58 0x501986ff58 0x501986fe40
pshipton commented 2 years ago

@0xdaryl @joransiu

dmitripivkine commented 2 years ago 
> !omrportlibraryglobaldata 0x0000005008621110
OMRPortLibraryGlobalData at 0x5008621110 {
  Fields for OMRPortLibraryGlobalData:
 0x0: void * corruptedMemoryBlock = !j9x 0x0000005C2328CA30 <--------------
 0x8: struct J9PortControlData control = !j9portcontroldata 0x0000005008621118
 0x30: struct J9NLSDataCache nls_data = !j9nlsdatacache 0x0000005008621140

0x5C2328C9D0 :  0000000000000000 0000000000000000 [ ................ ]
0x5C2328C9E0 :  0000000000000000 0000000000000050 [ ...............P ]
0x5C2328C9F0 :  63c32d0000000000 0000000000000000 [ c.-............. ]
0x5C2328CA00 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ]
0x5C2328CA10 :  0000000000000000 0000000000000050 [ ...............P ]
0x5C2328CA20 :  63c32d0000000000 0000000000000000 [ c.-............. ]
0x5C2328CA30 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ] <-----
0x5C2328CA40 :  0000000000000000 0000000000000050 [ ...............P ]
0x5C2328CA50 :  63c32d0000000000 0000000000000000 [ c.-............. ]
0x5C2328CA60 :  0000000000000050 63c32d0000000000 [ .......Pc.-..... ]
0x5C2328CA70 :  0000000000000000 0000000000000050 [ ...............P ]
0x5C2328CA80 :  63c32d0000000000 0000000000000000 [ c.-............. ]

Previous allocation with j9MemTag

0x5C2328C110 :  b1234567831ae912 0000000000000480 [ .#Eg............ ]
0x5C2328C120 :  0000000019934459 0000005008822db0 [ ......DY...P..-. ]
0x5C2328C130 :  0000005c2328c200 0000000000000000 [ ...\#(.......... ]
0x5C2328C140 :  0000000000000000 0000000000000000 [ ................ ]
...
0x5C2328C5A0 :  0000000000000000 0000000000000000 [ ................ ]
0x5C2328C5B0 :  b7654321855cebf4 0000000000000480 [ .eC!.\.......... ] <-- end of previous allocation
0x5C2328C5C0 :  0000000019934459 0000005008822db0 [ ......DY...P..-. ]
0x5C2328C5D0 :  0000000000000000 0000000000000000 [ ................ ]

contains data related to !j9class 0x0000005c2328c200 (java/util/stream/MatchOps$$Lambda$93/0x000000002328b830), particularly ramConstantPool = !j9x 0x0000005C2328C3C0

joransiu commented 2 years ago

@dmitripivkine : Just trying to make sure I interpret your analysis properly. Are you suspecting the ramConstantPool overflowed past the end of its allocated segment and corrupted the memory block @ 0x0000005C2328CA30? Does the data presented around 0x0000005C2328CA30 match waht we expect from constant pool i.e. the 0x50's and 0x53c32d0...0?

dmitripivkine commented 2 years ago

@dmitripivkine : Just trying to make sure I interpret your analysis properly. Are you suspecting the ramConstantPool overflowed past the end of its allocated segment and corrupted the memory block @ 0x0000005C2328CA30? Does the data presented around 0x0000005C2328CA30 match waht we expect from constant pool i.e. the 0x50's and 0x53c32d0...0?

No, I am not suggesting Constant Pool data is written out-of-bound. However there is common case for this type of corruption so I mentioned it. I did not look in details to CP data integrity. It looks ok at first look - footer is not damaged, data looks correct