search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

cmdLineTester_jvmtitests cma001 failing / crashing #11040

Closed pshipton closed 3 years ago

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_ppc64_aix_OMR/671/ cmdLineTester_jvmtitests_5 cmdLineTester_jvmtitests_7 cmdLineTester_jvmtitests_8

Testing: cma001
Test start time: 2020/10/28 18:16:40 Eastern Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_ppc64_aix_OMR_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xgcpolicy:metronome -Xcompressedrefs  -Xdump    -XX:ForceClassfileAsIntermediateData -agentlib:jvmtitest=test:ria001,args:V3 -agentlib:jvmtitest=test:rca001,args:V4 -agentlib:jvmtitest=test:cma001 -cp "/home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_ppc64_aix_OMR_testList_0/openjdk-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 20 milliseconds
Time spent executing: 9829 milliseconds
Test result: FAILED
Output from test:
>> Success condition was not found: [Return code: 0]

from last nightly build OpenJ9: fbbb2d6 OMR: e2fac34 OpenJDK8: bea7d86

from failing build OpenJ9: dd76180 OMR: 4d32dfd OpenJDK: 8425b0a

https://github.com/eclipse/openj9/compare/fbbb2d6...dd76180 Assuming it's not an OMR issue due to the next comment, where the same tests failed in a PR build without the latest OMR. The OpenJDK changes don't seem related. https://github.com/ibmruntimes/openj9-openjdk-jdk8/compare/bea7d86...8425b0a

The same tests are crashing on jdk11, similarly to the PR testing in the next comment. https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_ppc64_aix_OMR/676

[2020-10-28T23:04:21.081Z]  [ERR] mainSynchSignalHandler+0x704
[2020-10-28T23:04:21.081Z]  [ERR] +0x0
[2020-10-28T23:04:21.081Z]  [ERR] Unhandled exception
[2020-10-28T23:04:21.081Z]  [ERR] Type=Segmentation error vmState=0x0005ffff
[2020-10-28T23:04:21.081Z]  [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
[2020-10-28T23:04:21.081Z]  [ERR] Handler1=09001000A0F87278 Handler2=09001000A10FC580
[2020-10-28T23:04:21.081Z]  [ERR] R0=0000000000000000 R1=0000010021CAA060 R2=08001000A01A6A18 R3=0000010021CAA170
[2020-10-28T23:04:21.081Z]  [ERR] R4=0000000000000000 R5=0000002100000040 R6=000000000026259F R7=0000000000000000
[2020-10-28T23:04:21.081Z]  [ERR] R8=00000000154F01D7 R9=0000002100000040 R10=8000000000001032 R11=0000000000000000
[2020-10-28T23:04:21.081Z]  [ERR] R12=090000000EDA3B68 R13=0000010021CBA800 R14=0000010021CADC10 R15=0000010021CADAF8
[2020-10-28T23:04:21.081Z]  [ERR] R16=0000010021CADAFC R17=0000000000000080 R18=0000010021CADB00 R19=0000010021CABF77
[2020-10-28T23:04:21.081Z]  [ERR] R20=0000010021CABEF0 R21=0000010021CABEF4 R22=0000010021CAE7F8 R23=09001000A10FC580
[2020-10-28T23:04:21.081Z]  [ERR] R24=09001000A10FE350 R25=0000000000000018 R26=0000000000000018 R27=0000010021CAA170
[2020-10-28T23:04:21.081Z]  [ERR] R28=0000000000000000 R29=0000000000000000 R30=090000000F779E40 R31=09001000A0005420
[2020-10-28T23:04:21.081Z]  [ERR] IAR=090000000EDA3CE4 LR=090000000EDA3B8C MSR=A00000000000D032 CTR=090000000EDA3C80
[2020-10-28T23:04:21.081Z]  [ERR] CR=2220428820000001 FPSCR=0000000000000000 XER=2000000100000000
[2020-10-28T23:04:21.081Z]  [ERR] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2020-10-28T23:04:21.081Z]  [ERR] FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
[2020-10-28T23:04:21.081Z]  [ERR] FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
[2020-10-28T23:04:21.081Z]  [ERR] FPR6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR7 412e848000000000 (f: 0.000000, d: 1.000000e+06)
[2020-10-28T23:04:21.081Z]  [ERR] FPR8 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2020-10-28T23:04:21.081Z]  [ERR] FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
[2020-10-28T23:04:21.081Z]  [ERR] FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
[2020-10-28T23:04:21.081Z]  [ERR] FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
[2020-10-28T23:04:21.081Z]  [ERR] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2020-10-28T23:04:21.081Z]  [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
pshipton commented 3 years ago

Saw a similar problem in a jdk11 PR build, a crash in the same tests.

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_ppc64_aix_Personal/503/ OpenJ9: f702659 OMR: bb24584 OpenJDK: fd18c03

https://github.com/eclipse/openj9/compare/fbbb2d6...f702659

The problem appears to be caused by https://github.com/eclipse/openj9/pull/10930

DanHeidinga commented 3 years ago 
$ ~/jdks/adopt/oj9/jdk-15+36/Contents/Home/bin/java -Xjit:vmstate=0x0005ffff
vmState [0x5ffff]: {J9VMSTATE_JIT_CODEGEN} {Illegal optimization number}

I'm confused about how a Java language level change can be causing a crash during JIT codegen. Is the belief that this exposed an existing bug?

Additional info from the crash output:

[ERR] Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_ppc64_aix_OMR_testList_1/openjdkbinary/j2sdk-image/lib/compressedrefs/libj9jit29.so
 [ERR] Module_base_address=090000000E58B000
 [ERR] 
 [ERR] Method_being_compiled=com/ibm/jit/JITHelpers.getIntFromObject(Ljava/lang/Object;J)I
....
second crash:
[ERR] +0x0
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ff09
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033
 [ERR] Handler1=09001000A0F87278 Handler2=09001000A10FC580
...
ERR] printStackBacktrace__8TR_DebugFv+0x20705a (0x090000000EDA30A4 [libj9jit29.so+0x8180a4])
 [ERR] jitSignalHandler__FP13J9PortLibraryUiPvT3+0xe0 (0x090000000EDA2D64 [libj9jit29.so+0x817d64])
 [ERR] mainSynchSignalHandler+0x350 (0x090000000E454134 [libj9prt29.so+0x5c134])
 [ERR] (0x0000000000004BA4 [java+0x4ba4])
 [ERR] ---------------------------------------
 [ERR] Module=/usr/lib/libpthreads.a
 [ERR] Module_base_address=090000000056D000
 [ERR] 
 [ERR] Method_being_compiled=java/util/HashMap.resize()[Ljava/util/HashMap$Node;
 [ERR] Target=2_90_20201028_766 (AIX 7.1)
 [ERR] CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] monitor_enter+0x58 (0x090000000DF403DC [libj9thr29.so+0x83dc])
 [ERR] omrthread_monitor_enter+0x188 (0x090000000DF3ED6C [libj9thr29.so+0x6d6c])
 [ERR] omrthread_rwmutex_exit_read+0x34 (0x090000000DF4C798 [libj9thr29.so+0x14798])
 [ERR] exit_read__Q2_2J99RWMonitorFv+0x10 (0x090000000ED9FF14 [libj9jit29.so+0x814f14])
 [ERR] readReleaseClassUnloadMonitor__Q2_2J912MonitorTableFi@AF15_12+0x34 (0x090000000ED9FE38 [libj9jit29.so+0x814e38])
 [ERR] releaseClassUnloadMonitorAndAcquireVMaccessIfNeeded__11TR_J9VMBaseFPQ2_2TR11CompilationPb+0x68 (0x090000000EDD546C [libj9jit29.so+0x84a46c])
 [ERR] allocateCodeMemoryInner__Q2_2J913CodeGeneratorFUiT1PPUcb+0x19c (0x090000000E89B820 [libj9jit29.so+0x310820])
 [ERR] allocateCodeMemory__Q2_3OMR13CodeGeneratorFUiT1PPUcb+0x20 (0x090000000ED6F724 [libj9jit29.so+0x7e4724])
 [ERR] doBinaryEncoding__Q3_3OMR5Power13CodeGeneratorFv+0x164 (0x090000000ED6EE68 [libj9jit29.so+0x7e3e68])
 [ERR] performBinaryEncodingPhase__Q2_3OMR12CodeGenPhaseFPQ2_2TR13CodeGeneratorPQ2_2TR12CodeGenPhase+0x1d0 (0x090000000ED6E954 [libj9jit29.so+0x7e3954])
 [ERR] performAll__Q2_3OMR12CodeGenPhaseFv+0x174 (0x090000000ED52FF8 [libj9jit29.so+0x7c7ff8])
 [ERR] generateCode__Q2_3OMR13CodeGeneratorFv+0x198 (0x090000000ED5299C [libj9jit29.so+0x7c799c])
 [ERR] compile__Q2_3OMR11CompilationFv+0x1844 (0x090000000ECF88C8 [libj9jit29.so+0x76d8c8])
 [ERR] compile__Q2_2TR28CompilationInfoPerThreadBaseFP10J9VMThreadPQ2_2TR11CompilationP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRCQ2_2TR16SegmentAllocator+0x328 (0x090000000ECF552C [libj9jit29.so+0x76a52c])
 [ERR] wrappedCompile__Q2_2TR28CompilationInfoPerThreadBaseFP13J9PortLibraryPv+0xbb0 (0x090000000E7E83B4 [libj9jit29.so+0x25d3b4])
 [ERR] omrsig_protect+0x488 (0x090000000E45154C [libj9prt29.so+0x5954c])
 [ERR] compile__Q2_2TR28CompilationInfoPerThreadBaseFP10J9VMThreadP21TR_MethodToBeCompiledRQ2_2J917J9SegmentProvider+0x730 (0x090000000E7C6D34 [libj9jit29.so+0x23bd34])
 [ERR] processEntry__Q2_2TR24CompilationInfoPerThreadFR21TR_MethodToBeCompiledRQ2_2J917J9SegmentProvider+0x44c (0x090000000EE338D0 [libj9jit29.so+0x8a88d0])
 [ERR] processEntries__Q2_2TR24CompilationInfoPerThreadFv+0x1c8 (0x090000000EE361CC [libj9jit29.so+0x8ab1cc])
 [ERR] protectedCompilationThreadProc__FP13J9PortLibraryPQ2_2TR24CompilationInfoPerThread+0x504 (0x090000000EE35308 [libj9jit29.so+0x8aa308])
 [ERR] omrsig_protect+0x488 (0x090000000E45154C [libj9prt29.so+0x5954c])
 [ERR] compilationThreadProc__FPv+0x2c8 (0x090000000EE34A4C [libj9jit29.so+0x8a9a4c])
 [ERR] thread_wrapper+0x524 (0x090000000DF3C788 [libj9thr29.so+0x4788])
 [ERR] _pthread_body+0xf0 (0x0900000000570E14 [libpthreads.a+0x3e14])
 [ERR] ---------------------------------------
pshipton commented 3 years ago

@andrewcraik @gita-omr can somebody look at this please. I don't think it's intermittent so hopefully easy to track down. The problem is blocking https://github.com/eclipse/openj9/pull/10930 from being delivered, we had to back it out because of the failures.

DanHeidinga commented 3 years ago

Just for info - the agents in use in these tests do call Class#getCanonicalName, though I don't know how that would affect the JIT compilation. https://github.com/eclipse/openj9/blob/ce348df1cfc62f79fb9880af0a584065c824fe21/runtime/tests/jvmtitests/src/com/ibm/jvmti/tests/retransformationIncapableAgent/ria001.c#L193

gita-omr commented 3 years ago

I've seen such stack before. It was some intermittent failure that we were trying to debug. So most likely it's not caused by #10930 just seem to expose it in a more deterministic way. It was also on AIX.

gita-omr commented 3 years ago

We will focus on this one now.

gita-omr commented 3 years ago

Analyzed the core for https://github.com/eclipse/openj9/issues/11040#issuecomment-718279287

The actual crash seems to be in libpthreads (some function called from monitor_enter). We will need to get libpthreads.so from the failing machine to analyze further.

gita-omr commented 3 years ago

Alternatively, we can try and reproduce locally. But since it fails in many different ways, I think it's good to look at the core for #11040 (comment) since it seems to be most informative.

gita-omr commented 3 years ago

After getting libpthreads.a from the machine found out that we are crashing here:

0x90000000059b224 {libpthreads.a}{} +1469577      7d251829 lwarx     r9, r5, r3, 1
0x90000000059b228 {libpthreads.a}{} +1469578      65071000 oris      r7, r8, 0x1000
0x90000000059b22c {libpthreads.a}{} +1469579      2c890000 cmpwi     cr1, r9, 0
0x90000000059b230 {libpthreads.a}{} +1469580 *    40860014 bne       cr1, 0x90000000059b244 {libpthreads.a}{} +1469585 
0x90000000059b234 {libpthreads.a}{} +1469581 |    7ce5192d stwcx.    r7, r5, r3    <<<<<<<<<<<<<<<<<
0x90000000059b238 {libpthreads.a}{} +1469582 |    4c00012c isync     
0x90000000059b23c {libpthreads.a}{} +1469583 |    4d820020 beqlr     
0x90000000059b240 {libpthreads.a}{} +1469584 |    4bffffe4 b         0x90000000059b224 ^{libpthreads.a}{}

r3 should be pointing to _classUnloadMonitor inside J9::MonitorTable but it points to omrthread_rwmutex_is_writelocked which is a function.

It's practically impossible to find out from the core why _classUnloadMonitor is corrupted. Will need to run locally and try to reproduce.

gita-omr commented 3 years ago

Thanks @IBMJimmyk for continuing the investigation. Please note the problem happens with different builds but most frequently with the one from #10930. So it's probably the best to try to reproduce it locally with this particular build.

IBMJimmyk commented 3 years ago

I am currently looking at this issue and trying to reproduce the problem locally.

pshipton commented 3 years ago

This is set as a blocker since it's blocking https://github.com/eclipse/openj9/pull/10930 from being included in OpenJ9.

IBMJimmyk commented 3 years ago

I've been trying to use the failing build from this test: https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_ppc64_aix_Personal/503/

openjdk version "11.0.9-internal" 2020-10-20
OpenJDK Runtime Environment (build 11.0.9-internal+0-adhoc.jenkins.BuildJDK11ppc64aixPersonal)
Eclipse OpenJ9 VM (build HEAD-e15bce4ed9e, JRE 11 AIX ppc64-64-Bit Compressed References 20201028_772 (JIT enabled, AOT disabled)
OpenJ9   - e15bce4ed9e
OMR      - bb24584261c
JCL      - 31f23d7d216 based on jdk-11.0.9+11)

For comparison, I got a build off the adopt website that reverted the change the was triggering the problem. I'll refer to this as the "passing build" but it likely does have the same problem but just isn't being triggered for currently unknown reasons:

openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+4-202011292342)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-f10260975, JRE 11 AIX ppc64-64-Bit Compressed References 20201129_842 (JIT enabled, AOT disabled)
OpenJ9   - f10260975
OMR      - 1f46e75cf
JCL      - a34debfd22 based on jdk-11.0.10+4)

I was looking at the cma001 test inside cmdLineTester_jvmtitests_7. When I run cma001 on the passing build, it passed 10/10 times. When I run cma001 on the bad build it failed about 5/5 times so far. None of the failures have been quite the same as the one as Gita saw with a failure inside libpthreads. I've been trying out different options and it looks like this test fails under -Xint as well.

The Xint failure looks like this:

$ /team/jimmyk/gpu/JVMbuilds/OpenJ9-JDK11-ppc64_aix-20201028-105903/jdk/bin/java -XX:-EnableHCR -Xshareclasses:none -Xnoaot -Xms256m -Xmx256m -XX:+UseCompressedOops -Xint -Xgcpolicy:balanced -Xdump -XX:ForceClassfileAsIntermediateData -agentlib:jvmtitest=test:ria001,args:V3 -agentlib:jvmtitest=test:rca001,args:V4 -agentlib:jvmtitest=test:cma001 -cp "/team/jimmyk/gpu/jdk-tests/openj9/test/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033
Handler1=09001000A0A3F9B0 Handler2=09001000A0A16500
R0=0000000000000001 R1=000001001001BD40 R2=09001000A0A04380 R3=0000000000000000
R4=0000000000000001 R5=0000000000000000 R6=0000000000000000 R7=0000000000000000
R8=0000000013AA01CF R9=0000000013AA01CF R10=0B6B0451BF876840 R11=0000000000000000
R12=090000000B0C13DC R13=0000010010057800 R14=0005B58228DFC3B4 R15=09001000A09F9570
R16=000001001001C470 R17=09001000A0A860D0 R18=0000000000001308 R19=00000000000F0000
R20=0000000000000001 R21=0000000000000000 R22=00000100100E2018 R23=00000100100E0A80
R24=00000000000012C8 R25=00000100100E0A78 R26=00000000000015E0 R27=0000000000000000
R28=000001001019E278 R29=000000000000003A R30=0000000000000000 R31=0000010010070800
IAR=090000000B0C148C LR=090000000B0C13DC MSR=A00000000000D032 CTR=09000000005C8580
CR=2220084B20000008 FPSCR=8200000000000000 XER=2000000882000000
FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR7 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR13 0000000000000002 (f: 2.000000, d: 9.881313e-324)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/team/jimmyk/gpu/JVMbuilds/OpenJ9-JDK11-ppc64_aix-20201028-105903/jdk/lib/compressedrefs/libj9thr29.so
Module_base_address=090000000B0B9000
Target=2_90_20201028_772 (AIX 7.2)
CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
omrthread_monitor_enter+0x188 (0x090000000B0BFD6C [libj9thr29.so+0x6d6c])
omrthread_rwmutex_enter_write+0x34 (0x090000000B0CD898 [libj9thr29.so+0x14898])
managementClassLoadCounter+0x18 (0x090000000B7AE4BC [libjclse29.so+0x614bc])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
IPRA.$internalCreateRAMClassDone__FP10J9VMThreadP13J9ClassLoaderP10J9ROMClassUlP7J9ClassP6J9UTF8P21J9CreateRAMClassStateT5P15J9MemorySegment+0x8e8 (0x090000000ADA2FAC [libj9vm29.so+0x48fac])
IPRA.$internalCreateRAMClassFromROMClassImpl__FP10J9VMThreadP13J9ClassLoaderP10J9ROMClassUlP7J9ClassPP11J9ROMMethodliT5T4T5P21J9CreateRAMClassStateT2T5P8J9Module+0xd0c (0x090000000ADA3F10 [libj9vm29.so+0x49f10])
internalCreateRAMClassFromROMClass+0x2bc (0x090000000ADA06E0 [libj9vm29.so+0x466e0])
internalDefineClass+0x29c (0x090000000AE5A7A0 [libj9vm29.so+0x1007a0])
attemptDynamicClassLoad+0x138 (0x090000000AD9DF1C [libj9vm29.so+0x43f1c])
internalFindClassInModule+0x2de4 (0x090000000AD9C3E8 [libj9vm29.so+0x423e8])
internalFindClassString+0x148 (0x090000000AD9CEAC [libj9vm29.so+0x42eac])
bytecodeLoopCompressed+0x196e8 (0x090000000AF4B56C [libj9vm29.so+0x1f156c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
sendLoadClass+0x1a8 (0x090000000ADC26CC [libj9vm29.so+0x686cc])
internalFindClassInModule+0x21c0 (0x090000000AD9B7C4 [libj9vm29.so+0x417c4])
resolveClassRef+0x1ac (0x090000000ADDBC90 [libj9vm29.so+0x81c90])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveStaticMethodRefInto+0x200 (0x090000000ADDC6C4 [libj9vm29.so+0x826c4])
resolveStaticMethodRef+0x30 (0x090000000ADDCC14 [libj9vm29.so+0x82c14])
bytecodeLoopCompressed+0x4d70 (0x090000000AF36BF4 [libj9vm29.so+0x1dcbf4])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x090000000ADC0A14 [libj9vm29.so+0x66a14])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveStaticMethodRefInto+0x200 (0x090000000ADDC6C4 [libj9vm29.so+0x826c4])
resolveStaticMethodRef+0x30 (0x090000000ADDCC14 [libj9vm29.so+0x82c14])
bytecodeLoopCompressed+0x4d70 (0x090000000AF36BF4 [libj9vm29.so+0x1dcbf4])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x090000000ADC096C [libj9vm29.so+0x6696c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x090000000ADC0A14 [libj9vm29.so+0x66a14])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x090000000ADC096C [libj9vm29.so+0x6696c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
resolveStaticMethodRefInto+0x200 (0x090000000ADDC6C4 [libj9vm29.so+0x826c4])
resolveStaticMethodRef+0x30 (0x090000000ADDCC14 [libj9vm29.so+0x82c14])
bytecodeLoopCompressed+0x4d70 (0x090000000AF36BF4 [libj9vm29.so+0x1dcbf4])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpCheckCallin+0x74 (0x090000000ADB6818 [libj9vm29.so+0x5c818])
callVirtualObjectMethod+0x4c (0x090000000AE10E50 [libj9vm29.so+0xb6e50])
classPrepare+0x78 (0x0900000006C872FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x090000000B6C3EA8 [libj9jvmti29.so+0x8ea8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x090000000AFF344C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x090000000ADC0C1C [libj9vm29.so+0x66c1c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x090000000ADC0A14 [libj9vm29.so+0x66a14])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x090000000ADC096C [libj9vm29.so+0x6696c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x090000000ADC096C [libj9vm29.so+0x6696c])
resolveClassRef+0x3b4 (0x090000000ADDBE98 [libj9vm29.so+0x81e98])
bytecodeLoopCompressed+0x281a8 (0x090000000AF5A02C [libj9vm29.so+0x20002c])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
sendClinit+0x260 (0x090000000ADC3064 [libj9vm29.so+0x69064])
initializeImpl+0x3fc (0x090000000ADBF280 [libj9vm29.so+0x65280])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x090000000ADC049C [libj9vm29.so+0x6649c])
resolveStaticMethodRefInto+0x200 (0x090000000ADDC6C4 [libj9vm29.so+0x826c4])
resolveStaticMethodRef+0x30 (0x090000000ADDCC14 [libj9vm29.so+0x82c14])
bytecodeLoopCompressed+0x4d70 (0x090000000AF36BF4 [libj9vm29.so+0x1dcbf4])
(0x090000000ADC977C [libj9vm29.so+0x6f77c])
runCallInMethod+0x264 (0x090000000ADC63C8 [libj9vm29.so+0x6c3c8])
gpProtectedRunCallInMethod__FPv+0x48 (0x090000000ADB694C [libj9vm29.so+0x5c94c])
signalProtectAndRunGlue+0x28 (0x090000000ADB05AC [libj9vm29.so+0x565ac])
omrsig_protect+0x488 (0x090000000B0523CC [libj9prt29.so+0x593cc])
gpProtectAndRun+0xf8 (0x090000000ADB06FC [libj9vm29.so+0x566fc])
gpCheckCallin+0x114 (0x090000000ADB68B8 [libj9vm29.so+0x5c8b8])
callStaticVoidMethod+0x48 (0x090000000AE1200C [libj9vm29.so+0xb800c])
J9_CreateJavaVM+0x278 (0x090000000AD84FFC [libj9vm29.so+0x2affc])
JNI_CreateJavaVM_impl+0xa08 (0x090000000AD1FBAC [libjvm.so+0x26bac])
JNI_CreateJavaVM+0x1c4 (0x090000000ACE74C8 [libjvm.so+0x254c8])
JNI_CreateJavaVM+0x1c4 (0x090000000ACB04C8 [libjvm.so+0x254c8])
JavaMain+0xec (0x00000100000067D0 [java+0x67d0])
_pthread_body+0xe8 (0x0900000000599FEC [libpthreads.a+0x3fec])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2020/12/02 16:45:19 - please wait.
JVMDUMP032I JVM requested System dump using '/team/jimmyk/gpu/workspace/jvmtitests/core.20201202.164519.9175528.0001.dmp' in response to an event
JVMDUMP010I System dump written to /team/jimmyk/gpu/workspace/jvmtitests/core.20201202.164519.9175528.0001.dmp
JVMDUMP032I JVM requested Java dump using '/team/jimmyk/gpu/workspace/jvmtitests/javacore.20201202.164519.9175528.0002.txt' in response to an event
pshipton commented 3 years ago

Moving it to vm and also to the next milestone.

gita-omr commented 3 years ago

I would say the trace that @IBMJimmyk found and the original one are slightly related. Both are trying to enter some class loading related monitor..

IBMJimmyk commented 3 years ago

I tried making my own builds one with the latest openj9 and the other with the reverted change reverted. My expectations was for the run with the first build to work and the second build to fail. That's pretty much what I got but this time the second build failed via an assert.

Passing build:

openjdk version "11.0.10-internal" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11ppc64aixPersonal)
Eclipse OpenJ9 VM (build testBreak-f6fcc177cc1, JRE 11 AIX ppc64-64-Bit Compressed References 20201203_611 (JIT enabled, AOT disabled)
OpenJ9   - f6fcc177cc1
OMR      - db649084178
JCL      - b32e64a7cc9 based on jdk-11.0.10+5)

Failing build:

openjdk version "11.0.10-internal" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10-internal+0-adhoc.jenkins.BuildJDK11ppc64aixPersonal)
Eclipse OpenJ9 VM (build testBreak-7c89a5a4cf1, JRE 11 AIX ppc64-64-Bit Compressed References 20201203_612 (JIT enabled, AOT disabled)
OpenJ9   - 7c89a5a4cf1
OMR      - db649084178
JCL      - b32e64a7cc9 based on jdk-11.0.10+5)

Failing message (run with -Xint):

$ /team/jimmyk/gpu/JVMbuilds/OpenJ9-JDK11-ppc64_aix-20201203-122904-revert/jdk/bin/java -XX:+UseCompressedOops -Xint -Xgcpolicy:balanced -Xdump -XX:ForceClassfileAsIntermediateData -agentlib:jvmtitest=test:ria001,args:V3 -agentlib:jvmtitest=test:rca001,args:V4 -agentlib:jvmtitest=test:cma001 -cp "/team/jimmyk/gpu/jdk-tests/openj9/test/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
20:27:20.173 0x3000e700j9bcu(j9vm).128    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64_aix_Personal/openj9/runtime/bcutil/dynload.c:398: ((0 ))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2020/12/03 15:27:20 - please wait.
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
Handler1=09001000A07D74C8 Handler2=09001000A07ADFE8
R0=7C0802A6F8010010 R1=000001001001AAA0 R2=09001000A008F3A8 R3=7C0802A6F8010010
R4=FBE1FFF8607F0000 R5=00000000000022E2 R6=FBE1FFF8607F0000 R7=000000001413018D
R8=000000000413018D R9=0000000000000000 R10=0000000000000020 R11=0000000000000000
R12=0000000000000001 R13=0000010010057800 R14=0000000000000000 R15=0000000000000000
R16=0000000044550000 R17=000001001001BF3D R18=000001001001BF3C R19=000001001001BF28
R20=0000000000000000 R21=0900000005F263A0 R22=0000000000000001 R23=09001000A009EFE0
R24=0000000000000001 R25=0000000000000000 R26=09001000A009DBA8 R27=0000000000000000
R28=0000000000000000 R29=0000000000000000 R30=0000000000000020 R31=0000000000000010
???T>(?----------- Stack Backtrace -----------
malloc_common@AF104_87+0x48 (0x090000000002EBEC [libc.a+0x2ddec])
iconv_open_core+0x88c (0x09000000004B2BD0 [libiconv.a+0x1530])
iconv_open+0x28 (0x09000000004B36AC [libiconv.a+0x200c])
iconv_get+0x6c (0x0900000005E071F0 [libj9prt29.so+0x3f1f0])
convertMutf8ToPlatform+0x78 (0x0900000005E086BC [libj9prt29.so+0x406bc])
omrstr_convert+0x110 (0x0900000005E07694 [libj9prt29.so+0x3f694])
syslog_write+0x7c (0x0900000005E2D460 [libj9prt29.so+0x65460])
omrsyslog_write@AF8_1+0x50 (0x0900000005E2D974 [libj9prt29.so+0x65974])
IPRA.$writeSyslog+0x2d4 (0x0900000005E29838 [libj9prt29.so+0x61838])
j9nls_vprintf+0x17c (0x0900000005E29C40 [libj9prt29.so+0x61c40])
j9nls_printf+0x50 (0x0900000005E29A74 [libj9prt29.so+0x61a74])
triggerDumpAgents+0x58c (0x0900000005F147D0 [libj9dmp29.so+0x507d0])
doTriggerActionAssertDumpEvent+0x3c (0x0900000005F36DE0 [libj9trc29.so+0x6de0])
triggerHit+0x138 (0x0900000005F4E07C [libj9trc29.so+0x1e07c])
IPRA.$logTracePoint+0x318 (0x0900000005F5E63C [libj9trc29.so+0x2e63c])
doTracePoint@AF42_30+0x2ac (0x0900000005F60B70 [libj9trc29.so+0x30b70])
javaTrace+0x60 (0x0900000005F39C64 [libj9trc29.so+0x9c64])
IPRA.$searchClassInCPEntry+0xe4 (0x0900000005C9FC28 [libj9vm29.so+0x175c28])
IPRA.$searchClassInModule+0x94 (0x0900000005C9F918 [libj9vm29.so+0x175918])
findLocallyDefinedClass+0x1d0 (0x0900000005C9EE54 [libj9vm29.so+0x174e54])
IPRA.$callFindLocallyDefinedClass+0x1b0 (0x0900000005B6E454 [libj9vm29.so+0x44454])
attemptDynamicClassLoad+0xc0 (0x0900000005B6E824 [libj9vm29.so+0x44824])
internalFindClassInModule+0x2de4 (0x0900000005B6CD68 [libj9vm29.so+0x42d68])
internalFindClassString+0x148 (0x0900000005B6D82C [libj9vm29.so+0x4382c])
bytecodeLoopCompressed+0x19708 (0x0900000005D1F80C [libj9vm29.so+0x1f580c])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
sendLoadClass+0x1a8 (0x0900000005B92FCC [libj9vm29.so+0x68fcc])
internalFindClassInModule+0x21c0 (0x0900000005B6C144 [libj9vm29.so+0x42144])
resolveClassRef+0x1ac (0x0900000005BAC510 [libj9vm29.so+0x82510])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveStaticMethodRefInto+0x200 (0x0900000005BACF44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x0900000005BAD474 [libj9vm29.so+0x83474])
bytecodeLoopCompressed+0x4d70 (0x0900000005D0AE74 [libj9vm29.so+0x1e0e74])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x0900000005B91314 [libj9vm29.so+0x67314])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveStaticMethodRefInto+0x200 (0x0900000005BACF44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x0900000005BAD474 [libj9vm29.so+0x83474])
bytecodeLoopCompressed+0x4d70 (0x0900000005D0AE74 [libj9vm29.so+0x1e0e74])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x0900000005B9126C [libj9vm29.so+0x6726c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x0900000005B91314 [libj9vm29.so+0x67314])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x0900000005B9126C [libj9vm29.so+0x6726c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
resolveStaticMethodRefInto+0x200 (0x0900000005BACF44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x0900000005BAD474 [libj9vm29.so+0x83474])
bytecodeLoopCompressed+0x4d70 (0x0900000005D0AE74 [libj9vm29.so+0x1e0e74])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpCheckCallin+0x74 (0x0900000005B87198 [libj9vm29.so+0x5d198])
callVirtualObjectMethod+0x4c (0x0900000005BE1350 [libj9vm29.so+0xb7350])
classPrepare+0x78 (0x09000000059972FC [libjvmtitest.so+0x342fc])
jvmtiHookClassPrepare+0x104 (0x0900000006585DE8 [libj9jvmti29.so+0x8de8])
J9HookDispatch__FPP15J9HookInterfaceUlPv+0x1c8 (0x0900000005EA544C [libj9hookable29.so+0x44c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1618 (0x0900000005B9151C [libj9vm29.so+0x6751c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1410 (0x0900000005B91314 [libj9vm29.so+0x67314])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x0900000005B9126C [libj9vm29.so+0x6726c])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0x1368 (0x0900000005B9126C [libj9vm29.so+0x6726c])
resolveClassRef+0x3b4 (0x0900000005BAC718 [libj9vm29.so+0x82718])
bytecodeLoopCompressed+0x281c8 (0x0900000005D2E2CC [libj9vm29.so+0x2042cc])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
sendClinit+0x260 (0x0900000005B93964 [libj9vm29.so+0x69964])
initializeImpl+0x3fc (0x0900000005B8FB80 [libj9vm29.so+0x65b80])
classInitStateMachine__FP10J9VMThreadP7J9Class16J9ClassInitState+0xe98 (0x0900000005B90D9C [libj9vm29.so+0x66d9c])
resolveStaticMethodRefInto+0x200 (0x0900000005BACF44 [libj9vm29.so+0x82f44])
resolveStaticMethodRef+0x30 (0x0900000005BAD474 [libj9vm29.so+0x83474])
bytecodeLoopCompressed+0x4d70 (0x0900000005D0AE74 [libj9vm29.so+0x1e0e74])
(0x0900000005B9A1FC [libj9vm29.so+0x701fc])
runCallInMethod+0x264 (0x0900000005B96CC8 [libj9vm29.so+0x6ccc8])
gpProtectedRunCallInMethod__FPv+0x48 (0x0900000005B872CC [libj9vm29.so+0x5d2cc])
signalProtectAndRunGlue+0x28 (0x0900000005B80FAC [libj9vm29.so+0x56fac])
omrsig_protect+0x488 (0x0900000005E21BCC [libj9prt29.so+0x59bcc])
gpProtectAndRun+0xf8 (0x0900000005B810FC [libj9vm29.so+0x570fc])
gpCheckCallin+0x114 (0x0900000005B87238 [libj9vm29.so+0x5d238])
callStaticVoidMethod+0x48 (0x0900000005BE250C [libj9vm29.so+0xb850c])
J9_CreateJavaVM+0x278 (0x0900000005B55B7C [libj9vm29.so+0x2bb7c])
JNI_CreateJavaVM_impl+0xa08 (0x0900000005AEF92C [libjvm.so+0x2692c])
JNI_CreateJavaVM+0x1c4 (0x0900000005AB8248 [libjvm.so+0x25248])
JNI_CreateJavaVM+0x1c4 (0x0900000005A82248 [libjvm.so+0x25248])
JavaMain+0xec (0x00000100000067D0 [java+0x67d0])
_pthread_body+0xe8 (0x0900000000599FEC [libpthreads.a+0x3fec])

The assert appears to be triggering here: https://github.com/eclipse/openj9/blob/master/runtime/bcutil/dynload.c#L398

I think this is about as much as I can do so someone more familiar with the VM code will need to pick it up from here.

gita-omr commented 3 years ago

I agree. It's on VM's side now.

tajila commented 3 years ago

@JasonFengJ9 can you please take a look at this?

JasonFengJ9 commented 3 years ago

Sure, will look into it.

JasonFengJ9 commented 3 years ago

Run cmdLineTester_jvmtitests_5, cmdLineTester_jvmtitests_7 and cmdLineTester_jvmtitests_8 against latest AIX JDK 8/11, and no failure observed.

@pshipton did this failure occur in recent Adopt builds?

pshipton commented 3 years ago

This problem only occurs when https://github.com/eclipse/openj9/pull/10930 is applied.

JasonFengJ9 commented 3 years ago

Okay, since #10930 has been reverted, is this still a blocker?

pshipton commented 3 years ago

Kind of, it blocks #109030 from being merged.