search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

crash on 11.0.10 running jython hello world #12238

Closed pshipton closed 3 years ago

pshipton commented 3 years ago

This issue may already have been resolved, I haven't reproduced with 0.26.0 M1.

  1. Create a java file JythonIssue.java with below code 
    
import org.python.core.PyList;
import org.python.core.PyObject;
import org.python.core.PyString;
import org.python.util.InteractiveInterpreter;

public class JythonIssue {

public static void main(String[] arg) throws Throwable{ InteractiveInterpreter interp = new InteractiveInterpreter(); interp.exec("print('Hello Python World!')"); } }


2. Download jython-standalone-2.7.2.jar

3. compile the code

javac -classpath jython-standalone-2.7.2.jar:. JythonIssue.java

4. Run the test with below command

java -XshowSettings:vm -Dpython.options.showJavaExceptions=true -Dpython.options.includeJavaStackInExceptions=true -Dpython.options.showPythonProxyExceptions=true -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n -cp jython-standalone-2.7.2.jar:. JythonIssue


Expected output: (output is taken with JDK8 )

Listening for transport dt_socket at address: 8787 VM settings: Max. Heap Size (Estimated): 512.00M Ergonomics Machine Class: client Using VM: IBM J9 VM

Hello Python World!


Crash on AIX 7200-01-03-1720 with

openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9) Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.24.0, JRE 11 AIX ppc64-64-Bit Compressed References 20210120_896 (JIT enabled, AOT enabled) OpenJ9 - 345e1b09e OMR - 741e94ea8 JCL - 0a86953833 based on jdk-11.0.10+9)

Unhandled exception Type=Segmentation error vmState=0x00000000 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032 Handler1=09001000A1141AE8 Handler2=09001000A1117230 R0=382101007C0803A6 R1=000001001002A220 R2=09001000A008F3A8 R3=000001001001CFB0 R4=00000100107EEDE0 R5=E8410028E8010120 R6=090000000EA306C8 R7=0000000000000020 R8=0000010011709E60 R9=09001000A009EFC0 R10=09001000A009EFD0 R11=09001000A009EFD0 R12=0000000000000060 R13=0000010010057800 R14=000000000000001B R15=000000003000E700 R16=0000000000000000 R17=000001001002E8F0 R18=0000000000000001 R19=000001001002E9F0 R20=09001000A10F8AB0 R21=090000000EB0E7F8 R22=09001000A009EFE0 R23=0000000000000000 R24=09001000A0006D00 R25=0000010010608A20 R26=0000000000000500 R27=0000010010608F20 R28=0000000000000500 R29=09001000A009DBA8 R30=0000010010608A20 R31=0000010010608A30 IAR=0900000000047EFC LR=0900000000047894 MSR=A00000000000D032 CTR=09000000005C8480 CR=42A8442820000008 FPSCR=8200000000000000 XER=2000000882000000 FPR0 0000000000000001 (f: 1.000000, d: 4.940656e-324) FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18) FPR2 3ff0000000000000 (f: 0.000000, d: 1.000000e+00) FPR3 433000000004c141 (f: 311617.000000, d: 4.503600e+15) FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR5 4330000000000000 (f: 0.000000, d: 4.503600e+15) FPR6 4113050400000000 (f: 0.000000, d: 3.116170e+05) FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR8 433000000002a615 (f: 173589.000000, d: 4.503600e+15) FPR9 402da0ba1bf945c0 (f: 469321152.000000, d: 1.481392e+01) FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06) FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15) FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25) FPR13 0000000000000003 (f: 3.000000, d: 1.482197e-323) FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00) FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00) Module=/usr/lib/libc.a Module_base_address=0900000000000E00 Target=2_90_20210120_896 (AIX 7.2) CPU=ppc64 (16 logical CPUs) (0x200000000 RAM) ----------- Stack Backtrace ----------- free_common+0xc0 (0x090000000002E144 [libc.a+0x2d344]) omrmem_free_memory_basic+0x10 (0x090000000EBA91F4 [libj9prt29.so+0x2a1f4]) omrmem_free_memory+0x70 (0x090000000EBA6B54 [libj9prt29.so+0x27b54]) hashTableGrow+0x18c (0x090000000ED87910 [libj9gc29.so+0x4f910]) hashTableAdd+0x370 (0x090000000ED876D4 [libj9gc29.so+0x4f6d4]) addStringToInternTable14MM_StringTableFP10J9VMThreadP8J9Object+0x220 (0x090000000EF38564 [libj9gc29.so+0x200564]) j9gc_createJavaLangString+0x4fc (0x090000000EF3BD80 [libj9gc29.so+0x203d80]) resolveStringRef+0xd8 (0x090000000E95FF3C [libj9vm29.so+0x81f3c]) debugBytecodeLoopCompressed+0x251c0 (0x090000000EA81244 [libj9vm29.so+0x1a3244]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) resolveClassRef+0x3b4 (0x090000000E960718 [libj9vm29.so+0x82718]) debugBytecodeLoopCompressed+0x43fc8 (0x090000000EAA004C [libj9vm29.so+0x1c204c]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) resolveStaticMethodRefInto+0x200 (0x090000000E960F44 [libj9vm29.so+0x82f44]) resolveStaticMethodRef+0x30 (0x090000000E961474 [libj9vm29.so+0x83474]) debugBytecodeLoopCompressed+0x5c30 (0x090000000EA61CB4 [libj9vm29.so+0x183cb4]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000E944CCC [libj9vm29.so+0x66ccc]) resolveClassRef+0x3b4 (0x090000000E960718 [libj9vm29.so+0x82718]) debugBytecodeLoopCompressed+0x43fc8 (0x090000000EAA004C [libj9vm29.so+0x1c204c]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000E944CCC [libj9vm29.so+0x66ccc]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000E944CCC [libj9vm29.so+0x66ccc]) resolveClassRef+0x3b4 (0x090000000E960718 [libj9vm29.so+0x82718]) debugBytecodeLoopCompressed+0x43fc8 (0x090000000EAA004C [libj9vm29.so+0x1c204c]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000E944CCC [libj9vm29.so+0x66ccc]) resolveStaticMethodRefInto+0x200 (0x090000000E960F44 [libj9vm29.so+0x82f44]) resolveStaticMethodRef+0x30 (0x090000000E961474 [libj9vm29.so+0x83474]) debugBytecodeLoopCompressed+0x5c30 (0x090000000EA61CB4 [libj9vm29.so+0x183cb4]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) sendClinit+0x260 (0x090000000E947964 [libj9vm29.so+0x69964]) initializeImpl+0x3fc (0x090000000E943B80 [libj9vm29.so+0x65b80]) classInitStateMachineFP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000E944D9C [libj9vm29.so+0x66d9c]) resolveStaticMethodRefInto+0x200 (0x090000000E960F44 [libj9vm29.so+0x82f44]) resolveStaticMethodRef+0x30 (0x090000000E961474 [libj9vm29.so+0x83474]) debugBytecodeLoopCompressed+0x5c30 (0x090000000EA61CB4 [libj9vm29.so+0x183cb4]) (0x090000000E94E1FC [libj9vm29.so+0x701fc]) runCallInMethod+0x264 (0x090000000E94ACC8 [libj9vm29.so+0x6ccc8]) gpProtectedRunCallInMethodFPv+0x48 (0x090000000E93B2CC [libj9vm29.so+0x5d2cc]) signalProtectAndRunGlue+0x28 (0x090000000E934FAC [libj9vm29.so+0x56fac]) omrsig_protect+0x488 (0x090000000EBD8CCC [libj9prt29.so+0x59ccc]) gpProtectAndRun+0xf8 (0x090000000E9350FC [libj9vm29.so+0x570fc]) gpCheckCallin+0x114 (0x090000000E93B238 [libj9vm29.so+0x5d238]) callStaticVoidMethod+0x48 (0x090000000E99658C [libj9vm29.so+0xb858c]) JavaMain+0xd6c (0x0000010000007450 [java+0x7450]) _pthread_body+0xe8 (0x0900000000599FEC [libpthreads.a+0x3fec])

pshipton commented 3 years ago

@dmitripivkine @tajila fyi

I know there were some changes made related to String creation, although not specifically for this problem. Do you know anything else that may have fixed it? https://github.com/eclipse/openj9/pull/11668 https://github.com/eclipse/openj9/pull/11700

I have the core file in /team/peter/jython. The VM used is in aix-jdk-11

dmitripivkine commented 3 years ago

Are we sure this is not AIX 7.2.5 or this APAR is not installed? https://www.ibm.com/support/pages/apar/IJ30808

pshipton commented 3 years ago

Yes, it's 7200-01-03-1720 (aix72p7vm12).

pshipton commented 3 years ago

@llxia do we have any jython testing already, maybe in test buckets that don't get run by the OpenJ9 project? Otherwise we should consider adding some.

llxia commented 3 years ago

The closest that I can find is dacapo-jython. dacapo-jython is a perf test and it is disabled for 16+ due to https://github.com/eclipse/openj9/issues/11942

I think we should add the above test into our functional test set. I wonder if we can add existing jython tests as one of our external tests: https://github.com/jython/jython/tree/master/tests FYI @smlambert

pshipton commented 3 years ago

I've created https://github.com/eclipse/openj9/issues/12284 to track adding the hello world test.

pshipton commented 3 years ago

@tajila I see this was moved to the 0.27 release, but can somebody look at the core and try to confirm the problem has actually been fixed. It would be unfortunate to ship the 0.26 release still containing the problem, which could show up differently as well.

pshipton commented 3 years ago

@llxia looking at https://github.com/eclipse/openj9/issues/11942, it seems the dacapo-jython test uses an older version of jython (2.5.2). Does it make sense to update it to use the latest version where this problem was found (2.7.2)?

llxia commented 3 years ago

It looks like Dacapo benchmark bundles everything (including jython) into one jar. I am not sure if we can just update jython.

https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/perf/dacapo/build.xml#L30

tajila commented 3 years ago

@sharon-wang Can you please take a look at this

sharon-wang commented 3 years ago

Sure, I'll have a look.

sharon-wang commented 3 years ago

Ran with the instructions provided in the description on a few AIX 7.2 machines using the same version of JDK 11:

openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.24.0, JRE 11 AIX ppc64-64-Bit Compressed References 20210120_896 (JIT enabled, AOT enabled)
OpenJ9   - 345e1b09e
OMR      - 741e94ea8
JCL      - 0a86953833 based on jdk-11.0.10+9)

Confirming that I don't see the issue on AIX 7200-01-03-1720 machines. I do, however, see a different failure on an AIX 7200-05-01-2038 machine:

java -XshowSettings:vm -Dpython.options.showJavaExceptions=true -Dpython.options.includeJavaStackInExceptions=true -Dpython.options.showPythonProxyExceptions=true -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8788,server=y,suspend=n -cp jython-standalone-2.7.2.jar:.  JythonIssue
Listening for transport dt_socket at address: 8788
VM settings:
    Max. Heap Size (Estimated): 2.00G
    Using VM: Eclipse OpenJ9 VM

Unhandled exception
Type=Segmentation error vmState=0x00020019
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
Handler1=09001000A0CB1AE8 Handler2=09001000A0C89230
R0=0000000000000000 R1=0000010010024DA0 R2=0000000500000112 R3=000001001001CF50
R4=0000010011250AF8 R5=09001000A0C90398 R6=0000000000000000 R7=0000000000000000
R8=0000000000000000 R9=0000000000000000 R10=0000000000000004 R11=0000000000000000
R12=000000002288822B R13=0000010010057800 R14=0000000000000036 R15=000000003000E700
R16=0000000000000001 R17=0000010010029BC0 R18=0000000000000001 R19=0000010010029CC0
R20=0000000000000003 R21=0000000000000710 R22=000001001071F548 R23=0000000000000000
R24=0000000000000003 R25=000001001002F9A3 R26=00000100106A5AF0 R27=000001001071F530
R28=0000010010025760 R29=000000003000F110 R30=00000000000E1880 R31=000001001001CF50
IAR=FFFFFFFFFE395470 LR=0900000008A53414 MSR=A00000000000D032 CTR=0900000007540F14
CR=2888822000040000 FPSCR=8202000000000000 XER=0004000082020000
FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR6 4112cb8000000000 (f: 0.000000, d: 3.079360e+05)
FPR7 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR8 4020000000000000 (f: 0.000000, d: 8.000000e+00)
FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR13 0000000000000003 (f: 3.000000, d: 1.482197e-323)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Target=2_90_20210120_896 (AIX 7.2)
CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
flushNonAllocationCaches__22MM_EnvironmentDelegateFv+0x84 (0x0900000008A502C8 [libj9gc29.so+0x6c2c8])
flushNonAllocationCaches__22MM_EnvironmentStandardFv+0x18 (0x0900000008B8D09C [libj9gc29.so+0x1a909c])
flushCachesForGC__23GC_OMRVMThreadInterfaceFP18MM_EnvironmentBase+0x84 (0x0900000008A58C08 [libj9gc29.so+0x74c08])
flushCachesForGC__17GC_OMRVMInterfaceFP18MM_EnvironmentBase+0x90 (0x0900000008A58714 [libj9gc29.so+0x74714])
acquireExclusiveVMAccessForGC__18MM_EnvironmentBaseFP12MM_CollectorbT2+0x1b4 (0x0900000008A2AF78 [libj9gc29.so+0x46f78])
allocationRequestFailed__26MM_MemorySubSpaceSemiSpaceFP18MM_EnvironmentBaseP22MM_AllocateDescriptionQ2_17MM_MemorySubSpace14AllocationTypeP28MM_ObjectAllocationInterfaceP17MM_MemorySubSpaceT5+0x128 (0x0900000008B4702C [libj9gc29.so+0x16302c])
allocateTLH__24MM_MemorySubSpaceGenericFP18MM_EnvironmentBaseP22MM_AllocateDescriptionP28MM_ObjectAllocationInterfaceP17MM_MemorySubSpaceT4b+0x2d8 (0x0900000008B8FF9C [libj9gc29.so+0x1abf9c])
refresh__23MM_TLHAllocationSupportFP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x798 (0x0900000008BE263C [libj9gc29.so+0x1fe63c])
allocateFromTLH__23MM_TLHAllocationSupportFP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x178 (0x0900000008BE2C1C [libj9gc29.so+0x1fec1c])
allocateObject__25MM_TLHAllocationInterfaceFP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x504 (0x0900000008BE0B28 [libj9gc29.so+0x1fcb28])
allocateArrayletSpine__25MM_TLHAllocationInterfaceFP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x2c (0x0900000008BE0CF0 [libj9gc29.so+0x1fccf0])
OMR_GC_AllocateObject__FP12OMR_VMThreadP25MM_AllocateInitialization+0x204 (0x0900000008A63788 [libj9gc29.so+0x7f788])
J9AllocateIndexableObject+0xdc4 (0x0900000008A62268 [libj9gc29.so+0x7e268])
debugBytecodeLoopCompressed+0x4396c (0x09000000074F59F0 [libj9vm29.so+0x1c19f0])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendLoadClass+0x1a8 (0x090000000739CFCC [libj9vm29.so+0x68fcc])
internalFindClassInModule+0x27fc (0x0900000007376780 [libj9vm29.so+0x42780])
resolveClassRef+0x1ac (0x09000000073B6510 [libj9vm29.so+0x82510])
debugBytecodeLoopCompressed+0x43fc8 (0x09000000074F604C [libj9vm29.so+0x1c204c])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
resolveClassRef+0x3b4 (0x09000000073B6718 [libj9vm29.so+0x82718])
debugBytecodeLoopCompressed+0x43fc8 (0x09000000074F604C [libj9vm29.so+0x1c204c])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
resolveStaticMethodRefInto+0x200 (0x09000000073B6F44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x09000000073B7474 [libj9vm29.so+0x83474])
debugBytecodeLoopCompressed+0x5c30 (0x09000000074B7CB4 [libj9vm29.so+0x183cb4])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000739ACCC [libj9vm29.so+0x66ccc])
resolveClassRef+0x3b4 (0x09000000073B6718 [libj9vm29.so+0x82718])
debugBytecodeLoopCompressed+0x43fc8 (0x09000000074F604C [libj9vm29.so+0x1c204c])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000739ACCC [libj9vm29.so+0x66ccc])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000739ACCC [libj9vm29.so+0x66ccc])
resolveClassRef+0x3b4 (0x09000000073B6718 [libj9vm29.so+0x82718])
debugBytecodeLoopCompressed+0x43fc8 (0x09000000074F604C [libj9vm29.so+0x1c204c])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xdc8 (0x090000000739ACCC [libj9vm29.so+0x66ccc])
resolveStaticMethodRefInto+0x200 (0x09000000073B6F44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x09000000073B7474 [libj9vm29.so+0x83474])
debugBytecodeLoopCompressed+0x5c30 (0x09000000074B7CB4 [libj9vm29.so+0x183cb4])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x090000000739D964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000007399B80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000739AD9C [libj9vm29.so+0x66d9c])
resolveStaticMethodRefInto+0x200 (0x09000000073B6F44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x09000000073B7474 [libj9vm29.so+0x83474])
debugBytecodeLoopCompressed+0x5c30 (0x09000000074B7CB4 [libj9vm29.so+0x183cb4])
(0x09000000073A41FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x09000000073A0CC8 [libj9vm29.so+0x6ccc8])
gpProtectedRunCallInMethod__FPv+0x48 (0x09000000073912CC [libj9vm29.so+0x5d2cc])
signalProtectAndRunGlue+0x28 (0x090000000738AFAC [libj9vm29.so+0x56fac])
omrsig_protect+0x488 (0x0900000006C82CCC [libj9prt29.so+0x59ccc])
gpProtectAndRun+0xf8 (0x090000000738B0FC [libj9vm29.so+0x570fc])
gpCheckCallin+0x114 (0x0900000007391238 [libj9vm29.so+0x5d238])
callStaticVoidMethod+0x48 (0x09000000073EC58C [libj9vm29.so+0xb858c])
JavaMain+0xd6c (0x0000010000007450 [java+0x7450])
_pthread_body+0xe8 (0x09000000005B102C [libpthreads.a+0x402c])
pshipton commented 3 years ago

I did manage to get a crash on 7200-01-03-1720 (aix72p7vm12), although not right away. The core is still available if you want to look at it.

On AIX 7200-05-01 there is the known problem https://www.ibm.com/support/pages/apar/IJ30808 so I'm not sure if the crash is due to this, @dmitripivkine may know.

dmitripivkine commented 3 years ago

this is problematic memory decoding from the core:

0x1001001BEA0 :  5b5b5b5b00000000 0000000000000060 [ [[[[...........` ] <-- correct AIX header
0x1001001BEB0 :  b12345670f11b18a 0000000000000019 [ .#Eg............ ] <-- correct JVM header(JVMTI Allocate())
0x1001001BEC0 :  090000000f3c0974 09001000a10f5230 [ .....<.t......R0 ]
0x1001001BED0 :  4c6a6176612f6c61 6e672f537472696e [ Ljava/lang/Strin ]
0x1001001BEE0 :  674c6174696e313b 00dddddddddddddd [ gLatin1;........ ]
0x1001001BEF0 :  b76543210957b78c 0000000000000019 [ .eC!.W.......... ] <-- correct JVM footer
0x1001001BF00 :  090000000f3c0974 09001000a10f5230 [ .....<.t......R0 ]
0x1001001BF10 :  00000100100627e0 0000000000000060 [ ......'........` ]
0x1001001BF20 :  5b5b5b5b00000000 0000000000000058 [ [[[[...........X ] <-- correct AIX header
0x1001001BF30 :  b12345670f11b004 0000000000000017 [ .#Eg............ ] <-- corrupted JVM header
0x1001001BF40 :  0000000000000020 000000003000e700 [ ....... ....0... ]
0x1001001BF50 :  000000008024a7f8 2f7574696c2f5552 [ .....$../util/UR ]
0x1001001BF60 :  4c5574696c3b00dd b76543210957b61a [ LUtil;...eC!.W.. ] <-- corrupted JVM footer
0x1001001BF70 :  0000000000000017 000000008024a818 [ .............$.. ]
0x1001001BF80 :  0000000000000024 000000003000e700 [ .......$....0... ]
0x1001001BF90 :  000000003000e700 00000000802b7f58 [ ....0........+.X ]
0x1001001BFA0 :  00000000802b7f58 00000000802b7f7c [ .....+.X.....+.| ] <-- corrupted AIX header (missed 5b5b5b5b)
0x1001001BFB0 :  b12345670e836763 0000000000000180 [ .#Eg..gc........ ] <-- correct JVM header
0x1001001BFC0 :  090000000eaedf54 09001000a10f5260 [ .......T......R` ]
0x1001001BFD0 :  00000000802b8708 0000000000000004 [ .....+.......... ]
0x1001001BFE0 :  000000003000e700 00000000300cc108 [ ....0.......0... ]
0x1001001BFF0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C000 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C010 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C020 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C030 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C040 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C050 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C060 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C070 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C080 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C090 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0A0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0B0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0C0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0D0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0E0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C0F0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C100 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C110 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C120 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C130 :  0000000000000000 000001001143e658 [ .............C.X ]
0x1001001C140 :  000001001143e680 0000000000000000 [ .....C.......... ]
0x1001001C150 :  000000003000e700 00000000fffc1d80 [ ....0........... ] <-- corrupted JVM footer (missed b7654321)
0x1001001C160 :  00000000ffee3d10 00000000ffee3d10 [ ......=.......=. ]
0x1001001C170 :  000000003000e700 00000000fffc1d80 [ ....0........... ]
0x1001001C180 :  00000000fffc1d84 00000000ffee3d10 [ ..............=. ]
0x1001001C190 :  00000000fffd806c 0000000000000000 [ .......l........ ]
0x1001001C1A0 :  0000000000000000 00000000fffd8652 [ ...............R ]
0x1001001C1B0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C1C0 :  0000000000000000 00000100113b5030 [ .............;P0 ]
0x1001001C1D0 :  09001000a0f0bb28 09001000a0f0bb40 [ .......(.......@ ]
0x1001001C1E0 :  0000000000000000 0000004000000000 [ ...........@.... ]
0x1001001C1F0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C200 :  5b5b5b5b00000000 0000000000000070 [ [[[[...........p ] <-- correct AIX header
0x1001001C210 :  b12345670f11cd1f 000000000000002c [ .#Eg..........., ] <-- correct JVM header (JVMTI Allocate())
0x1001001C220 :  090000000f3c0974 09001000a10f5230 [ .....<.t......R0 ]
0x1001001C230 :  4c6a6176612f7574 696c2f7a69702f49 [ Ljava/util/zip/I ]
0x1001001C240 :  6e666c6174657224 496e666c61746572 [ nflater$Inflater ]
0x1001001C250 :  5a53747265616d52 65663b00dddddddd [ ZStreamRef;..... ]
0x1001001C260 :  b76543210957cb29 000000000000002c [ .eC!.W.)......., ] <-- correct JVM footer
0x1001001C270 :  090000000f3c0974 09001000a10f5230 [ .....<.t......R0 ]
0x1001001C280 :  5b5b5b5b00000000 0000000000000070 [ [[[[...........p ] <-- correct AIX header

> !findheader 0x1001001BED0
Searching memory allocation header for 0x000001001001BED0
Found memory allocation header, !j9x 0x1001001bed0,0x19
J9MemTag at 0x1001001beb0 {
    U_32 eyeCatcher = 0xb1234567;
    U_32 sumCheck = 0xf11b18a;
    UDATA allocSize = 0x19;
    char* callSite = /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/jvmti/jvmtiClass.c:375;
    struct OMRMemCategory* category = !omrmemcategory 0x9001000a10f5230 (JVMTI Allocate());
}

> !j9memtag 0x1001001BFB0
J9MemTag at 0x1001001bfb0 {
  Fields for J9MemTag:
    0x0: U32 eyeCatcher = 0xB1234567 (2971878759)
    0x4: U32 sumCheck = 0x0E836763 (243492707)
    0x8: U64 allocSize = 0x0000000000000180 (384)
    0x10: const U8* callSite = !j9x 0x090000000EAEDF54 // "/home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/vm/jnicsup...."
    0x18: struct OMRMemCategory* category = !omrmemcategory 0x09001000A10F5260
}
> !omrmemcategory 0x09001000A10F5260
OMRMemCategory at 0x9001000a10f5260 {
  Fields for OMRMemCategory:
    0x0: const U8* name = !j9x 0x090000000E8C8090 // "JNI"
    0x8: const U32 categoryCode = 0x00000013 (19)
    0x10: U64 liveBytes = 0x000000000000B508 (46344)
    0x18: U64 liveAllocations = 0x0000000000000059 (89)
    0x20: const U32 numberOfChildren = 0x00000000 (0)
    0x28: const U32* children = !j9x 0x0000000000000000
}

> !findheader 0x1001001C240
Searching memory allocation header for 0x000001001001C240
J9MemTag check failed at 1001001c150: Corrupt memory section found :Wrong eyecatcher. Expected 0xb7654321 but was 0x000001001001C150
Found memory allocation header, !j9x 0x1001001c230,0x2c
J9MemTag at 0x1001001c210 {
    U_32 eyeCatcher = 0xb1234567;
    U_32 sumCheck = 0xf11cd1f;
    UDATA allocSize = 0x2c;
    char* callSite = /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/jvmti/jvmtiClass.c:375;
    struct OMRMemCategory* category = !omrmemcategory 0x9001000a10f5230 (JVMTI Allocate());
}

So if this is not general AIX malloc issue it might relate to JNI/JVMTI code writing out-of bound (based on all area is filled by small memory allocations from JNI/JVMTI)

dmitripivkine commented 3 years ago

if core from crash mentioned in https://github.com/eclipse/openj9/issues/12238#issuecomment-807053153 has been preserved I can take a look too

sharon-wang commented 3 years ago

@dmitripivkine I've sent over some info on how to access the cores from https://github.com/eclipse/openj9/issues/12238#issuecomment-807053153.

dmitripivkine commented 3 years ago

I looked to one of cores core.20210325.101133.8782210.0001.dmp and it is memory corruption again:

mainSynchSignalHandler(??, ??, ??), line 1066 in "omrsignal.c"
.() at 0xfffffffffe395470
UnfinalizedObjectBuffer.MM_UnfinalizedObjectBuffer::flush(MM_EnvironmentBase*)(??, ??), line 70 in "UnfinalizedObjectBuffer.cpp"
EnvironmentDelegate.MM_EnvironmentDelegate::flushNonAllocationCaches()(??), line 149 in "EnvironmentDelegate.cpp"
MM_EnvironmentStandard::flushNonAllocationCaches()(??), line 652 in "EnvironmentBase.hpp"
OMRVMThreadInterface.GC_OMRVMThreadInterface::flushCachesForGC(MM_EnvironmentBase*)(??), line 44 in "OMRVMThreadInterface.cpp"
OMRVMInterface.GC_OMRVMInterface::flushCachesForGC(MM_EnvironmentBase*)(??), line 132 in "OMRVMInterface.cpp"
EnvironmentBase.MM_EnvironmentBase::acquireExclusiveVMAccessForGC(MM_Collector*,bool,bool)(??, ??, ??, ??), line 450 in "EnvironmentBase.cpp"
MM_MemorySubSpaceSemiSpace::allocationRequestFailed(MM_EnvironmentBase*,MM_AllocateDescription*,MM_MemorySubSpace::AllocationType,MM_ObjectAllocationInterface*,MM_MemorySubSpace*,MM_MemorySubSpace*)(??, ??, ??, ??, ??, ??, ??), line 99 in "MemorySubSpaceSemiSpace.cpp"
.......

> !gc_environment 0x000001001124EA20
GC_Environment at 0x1001124ea20 {
  Fields for GC_Environment:
        0x0: class MM_MarkJavaStats _markJavaStats = !mm_markjavastats 0x000001001124EA20
        0x98: class MM_ScavengerJavaStats _scavengerJavaStats = !mm_scavengerjavastats 0x000001001124EAB8
        0x118: class MM_ReferenceObjectBuffer* _referenceObjectBuffer = !mm_referenceobjectbufferstandard 0x000001001001CEB0
        0x120: class MM_UnfinalizedObjectBuffer* _unfinalizedObjectBuffer = ! 0x000001001001CF50
        0x128: class MM_OwnableSynchronizerObjectBuffer* _ownableSynchronizerObjectBuffer = !mm_ownablesynchronizerobjectbufferstandard 0x00000100111D6AF0
}

both _referenceObjectBuffer and _unfinalizedObjectBuffer are corrupted

0x1001001CE80 :  5b5b5b5b00000000 0000000000000088 [ [[[[............ ]
0x1001001CE90 :  b12345670937e533 0000000000000048 [ .#Eg.7.3.......H ] <-- JVM header for _referenceObjectBuffer
0x1001001CEA0 :  0900000008d3804c 09001000a0c6ffc0 [ .......L........ ]
0x1001001CEB0 :  000001001001cf60 0900000008d38000 [ .......`........ ] <--  !mm_referenceobjectbufferstandard 0x000001001001CEB0
0x1001001CEC0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001CED0 :  000001001001cf60 09001000a0cb55b0 [ .......`......U. ]
0x1001001CEE0 :  000001001001cf60 0000000000000044 [ .......`.......D ]
0x1001001CEF0 :  0000000000000000 09001000a0cb55b0 [ ..............U. ] <-- JVM footer missed at 0x1001001CEF8
0x1001001CF00 :  0000000000000048 09001000a0cb55b0 [ .......H......U. ]
0x1001001CF10 :  09001000a0c6ffc0 09000000004af220 [ .............J.  ]
0x1001001CF20 :  000001001001cfa0 000000000000027c [ ...............| ]
0x1001001CF30 :  09001000a0c90400 0000000000000001 [ ................ ] <-- JVM header missed for _unfinalizedObjectBuffer
0x1001001CF40 :  000001001001d740 000001001001d740 [ .......@.......@ ]
0x1001001CF50 :  000001001001d110 000001001001ddc0 [ ................ ] <-- corrupted !mm_unfinalizedobjectbufferstandard 0x1001001CF50
0x1001001CF60 :  000001001001d0a0 0000000000000000 [ ................ ]
0x1001001CF70 :  090000000748e688 0000000000000044 [ .....H.........D ]

Crash occur an attempt to use !mm_unfinalizedobjectbufferstandard 0x1001001CF50 due corruption of the data in memory Also there is another corrupted memory area around 0x1001001c820

pshipton commented 3 years ago

Btw when the test case doesn't crash on 11.0.10, it seems to just hang.

smlambert commented 3 years ago

re: https://github.com/eclipse/openj9/issues/12238#issuecomment-806026763, @llxia - I see this is being added into sanity.functional, but it would also be well-suited to be included as a startup test in sanity.external (as we have always considered creating a startup test for each of the applications in the external directory, see https://github.com/AdoptOpenJDK/openjdk-tests/tree/master/external#technical-goals - Startup-only testing, targets that would not require Docker on a test machine in order to be run).

dmitripivkine commented 3 years ago

This is from another Sharon's core core.20210325.123010.21627138.0001.dmp: it crashed an attempt to flush corrupted buffer !mm_unfinalizedobjectbufferstandard 0x000001001001CF50:

0x1001001CF00 :  0000000000000000 09001000a0011de0 [ ................ ]
0x1001001CF10 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001CF20 :  00000000006e1f3c 0000000000000000 [ .....n.<........ ]
0x1001001CF30 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001CF40 :  0000000000000000 0000000230033520 [ ............0.5  ]
0x1001001CF50 :  ffffffffffffffff 000001001001d3d0 [ ................ ] <-- !mm_unfinalizedobjectbufferstandard
0x1001001CF60 :  ffffffffffffffff 0000000000000000 [ ................ ]
0x1001001CF70 :  ffffffffffffffff ffffffffffffffff [ ................ ]
0x1001001CF80 :  ffffffffffffffff ffffffffffffffff [ ................ ]
0x1001001CF90 :  ffffffffffffffff ffffffffffffffff [ ................ ]
0x1001001CFA0 :  ffffffffffffffff ffffffffffffffff [ ................ ]
0x1001001CFB0 :  ffffffffffffffff 0000010010057800 [ ..............x. ]
0x1001001CFC0 :  000001001019e8e0 0000000000000000 [ ................ ]
0x1001001CFD0 :  0000000044550000 000001001001f0dd [ ....DU.......... ]
0x1001001CFE0 :  000001001001f0dc 0000000000000000 [ ................ ]
0x1001001CFF0 :  0000000000000001 09001000a0c7d738 [ ...............8 ]

What if it is overwritten by owner's of previous buffer? Looking in memory backwards for first allocation tag:

> !j9memtag 0x1001001C490
J9MemTag at 0x1001001c490 {
  Fields for J9MemTag:
        0x0: U32 eyeCatcher = 0xB1234567 (2971878759)
        0x4: U32 sumCheck = 0x06B1AE8F (112307855)
        0x8: U64 allocSize = 0x00000000000002C8 (712)
        0x10: const U8* callSite = !j9x 0x0900000007543ED0 // "/home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/vm/jnicsup...."
        0x18: struct OMRMemCategory* category = !omrmemcategory 0x09001000A0C70260
}
> !omrmemcategory 0x09001000A0C70260
OMRMemCategory at 0x9001000a0c70260 {
  Fields for OMRMemCategory:
        0x0: const U8* name = !j9x 0x0900000006B4D090 // "JNI"
        0x8: const U32 categoryCode = 0x00000013 (19)
        0x10: U64 liveBytes = 0x000000000000B508 (46344)
        0x18: U64 liveAllocations = 0x0000000000000059 (89)
        0x20: const U32 numberOfChildren = 0x00000000 (0)
        0x28: const U32* children = !j9x 0x0000000000000000
}

0x1001001C480 :  5b5b5b5b00000000 0000000000000308 [ [[[[............ ] <--- AIX malloc eyecatchger, size 308 bytes
0x1001001C490 :  b123456706b1ae8f 00000000000002c8 [ .#Eg............ ] <--- JVM header eyecatcgher
0x1001001C4A0 :  0900000007543ed0 09001000a0c70260 [ .....T>........` ]
0x1001001C4B0 :  0000000000000010 0000004000000000 [ ...........@.... ]
0x1001001C4C0 :  00000000000003b0 00000000011bd228 [ ...............( ]
0x1001001C4D0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C4E0 :  0000000000000000 0000000000000000 [ ................ ]
0x1001001C4F0 :  000001001001d2e0 000000000000001f [ ................ ]
0x1001001C500 :  000001001001c5e0 0000000000000004 [ ................ ]
0x1001001C510 :  000001001001c5f0 09001000a0511078 [ .............Q.x ]
0x1001001C520 :  0000000000000020 09001000a0c8a550 [ ....... .......P ]
0x1001001C530 :  000000000000000c 09001000a00913b8 [ ................ ]
0x1001001C540 :  000001001001c620 0000000000000021 [ ....... .......! ]
0x1001001C550 :  0a000100077e8bb4 0000000000000014 [ .....~.......... ]
0x1001001C560 :  00000000300de700 09001000a00913b8 [ ....0........... ]
0x1001001C570 :  0000000000000022 09000000005b1076 [ .......".....[.v ]
0x1001001C580 :  000001001001c640 0000000000000010 [ .......@........ ]
0x1001001C590 :  000001001001c640 0000000000000023 [ .......@.......# ]
0x1001001C5A0 :  000001001001c9b8 09001000a00913b8 [ ................ ]
0x1001001C5B0 :  000001001001c670 09001000a00913b8 [ .......p........ ]
0x1001001C5C0 :  000001001001c670 0000000000000000 [ .......p........ ]
0x1001001C5D0 :  000001001001ca30 09001000a00913b8 [ .......0........ ]
0x1001001C5E0 :  0000000000000082 09001000a00913b8 [ ................ ]
0x1001001C5F0 :  0000000000008440 00000100101a5cf0 [ .......@......\. ]
0x1001001C600 :  0000000000000082 0000000000000009 [ ................ ]
0x1001001C610 :  09001000a0005630 0000000000001010 [ ......V0........ ]
0x1001001C620 :  000001001001c6d0 000001001001d7f4 [ ................ ]
0x1001001C630 :  0000000000000082 0000000000000000 [ ................ ]
0x1001001C640 :  0000000000008440 00000100101a5cf0 [ .......@......\. ]
0x1001001C650 :  0000000000000082 0000000000000009 [ ................ ]
0x1001001C660 :  09001000a0005630 0000000000000002 [ ......V0........ ]
0x1001001C670 :  000001001001c6e0 33882454740a0000 [ ........3.$Tt... ]
0x1001001C680 :  0900000000039a68 0000010011db8a10 [ .......h........ ]
0x1001001C690 :  0900000000031c00 0000000000000000 [ ................ ]
0x1001001C6A0 :  0000000000008440 00000100101a5cf0 [ .......@......\. ]
0x1001001C6B0 :  0000000000000082 0000000000000009 [ ................ ]
0x1001001C6C0 :  09001000a0005630 0000000000000002 [ ......V0........ ]
0x1001001C6D0 :  000001001001c760 09001000a00913b8 [ .......`........ ]
0x1001001C6E0 :  000001001001c760 09001000a00913b8 [ .......`........ ]
0x1001001C6F0 :  0900000000038d28 09001000a00913b8 [ .......(........ ]
0x1001001C700 :  0000000000001000 09001000a00913b8 [ ................ ]
0x1001001C710 :  0000010011ddb6b0 0000580000005800 [ ..........X...X. ]
0x1001001C720 :  0000000000000000 0000000000001020 [ ...............  ]
0x1001001C730 :  0000000000000000 000000001483022f [ .............../ ]
0x1001001C740 :  000000001483022f 0000010011db8a10 [ ......./........ ]
0x1001001C750 :  09001000a0005630 0000000000000002 [ ......V0........ ]
0x1001001C760 :  000001001001c7f0 09001000a0006924 [ ..............i$ ]
0x1001001C770 :  090000000003cc10 0000000000000000 [ ................ ] <--- JVM footer expected to be here
0x1001001C780 :  000001001001d960 09001000a00913b8 [ .......`........ ]
0x1001001C790 :  000001001001c800 0000000000000000 [ ................ ] <--- should end here
0x1001001C7A0 :  09001000a0005630 09001000a0510b00 [ ......V0.....Q.. ]
0x1001001C7B0 :  000001001001c820 09001000a0005630 [ ....... ......V0 ]
0x1001001C7C0 :  000001001001c830 0000000000000000 [ .......0........ ]
0x1001001C7D0 :  0900000000393ae8 09001000a0005630 [ .....9:.......V0 ]
0x1001001C7E0 :  0000000000000001 0000010011db8990 [ ................ ]
0x1001001C7F0 :  000001001001c880 4488224800000000 [ ........D."H.... ]
0x1001001C800 :  000001001001c870 09001000a00913b8 [ .......p........ ]

and so on, reach 0x1001001CF50 eventually For first look data looks contiguous, there is a slim chance this allocation is written out-of-bound. Unfortunately due long path the owner of allocation is truncated:

// "/home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-openj9/workspace/build/src/openj9/runtime/vm/jnicsup...."

We can increase size of buffer to prevent truncation and try again. However I have no explanation why mm_unfinalizedobjectbufferstandard is a victim repeatedly.

dmitripivkine commented 3 years ago

... or maybe allocation from comment above is not involved - I am looking to another core and memory layout is very similar but this 0x2c8 bytes allocation is there and not out-of-bound obviously. Also with closer look this allocation has a distinct structure and this pattern does not repeat over reverted size

sharon-wang commented 3 years ago

After running 50+ iterations of the sample program on AIX 7200-01-03-1720 machines, still not able to reproduce and always hangs right after printing:

Listening for transport dt_socket at address: 8788
VM settings:
    Max. Heap Size (Estimated): 2.00G
    Using VM: Eclipse OpenJ9 VM

so, "Hello Python World!" is never seen.

pshipton commented 3 years ago

I'm closing this. https://github.com/eclipse-openj9/openj9/issues/12284 added jython testing into the builds, and this test is passing.