Closed llxia closed 6 years ago
@fjeremic - can someone look at this as soon as possible? https://github.com/eclipse/openj9/issues/2821
@fjeremic - can someone look at this as soon as possible? #2821
@dchopra001 can you please take a look at this one?
This issue is no longer reproducible.
I discussed with Lan, and this failure is no longer seen in recent builds: https://ci.eclipse.org/openj9/view/Test/job/Test-extended.functional-JDK11-linux_390-64_cmprssptrs/18/tapResults/
Just to be sure, I re-ran this threadMXBeanTimersTest_3
test several times to try and reproduce the issue. After 160 iterations, I have 0 reproductions of this issue.
Based on the above information, I don't think this is an issue anymore. So we can close this.
@llxia FYI...
Noticed this test failed in ppcle: https://ci.eclipse.org/openj9/view/Test/job/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/19/tapResults/
===============================================
Running test threadMXBeanTimersTest_3 ...
===============================================
threadMXBeanTimersTest_3 Start Time: Mon Sep 17 09:14:04 2018 Epoch Time (ms): 1537175644763
test with Mode610
[IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/resources/excludes/latest_exclude_SE110.txt
...
... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
...
Unhandled exception
Type=Segmentation error vmState=0x00030000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00003FFF8CF0CA70 Handler2=00003FFF8CC9D2B0
R0=00003FFF87D7E6BC R1=00003FFF6D04B0B0 R2=00003FFF8D047000 R3=0000000000408200
R4=00003FFF6D04B388 R5=0000000000000000 R6=0000000000000000 R7=00003FFF6D04B140
R8=00003FFF6D04B148 R9=00003FFF8CF0CFE0 R10=0000000000000001 R11=0000000000000006
R12=00003FFF8CF0CFE0 R13=00003FFF6D056900 R14=0000000000407990 R15=0000000000408200
R16=00003FFF67710038 R17=FFFFFFFFFFFFFFFF R18=0000000000000080 R19=00003FFF6D3BF8B8
R20=0000000000000000 R21=0000000000014F00 R22=000000000000111A R23=0000000000000000
R24=000000000040B160 R25=0000000000000000 R26=00000000004108F0 R27=0000000000000000
R28=00003FFF6D04B490 R29=0000000000000000 R30=00003FFF6D04B4A8 R31=00003FFF6D04B388
NIP=00003FFF8CF0CFE8 MSR=800000010280F033 ORIG_GPR3=00003FFF84CD0578 CTR=00003FFF8CF0CFE0
LINK=00003FFF87D7E4CC XER=0000000020000000 CCR=0000000048004444 SOFTE=0000000000000001
TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
FPR0 00000000003217d0 (f: 3282896.000000, d: 1.621966e-317)
FPR1 40528078e43bb0a0 (f: 3829117184.000000, d: 7.400738e+01)
FPR2 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
FPR3 3c73e00000000000 (f: 0.000000, d: 1.723881e-17)
FPR4 402a56ef8ec92280 (f: 2395546112.000000, d: 1.316980e+01)
FPR5 3ff61b97a0000000 (f: 2684354560.000000, d: 1.381736e+00)
FPR6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR8 00000000004080ba (f: 4227258.000000, d: 2.088543e-317)
FPR9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR10 0061006500700065 (f: 7340133.000000, d: 7.565937e-307)
FPR11 37f0003a00000000 (f: 0.000000, d: 2.938898e-39)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 402afc7c814a849f (f: 2169144576.000000, d: 1.349314e+01)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdkbinary/j2sdk-image/lib/compressedrefs/libj9vm29.so
Module_base_address=00003FFF8CE60000
Target=2_90_20180917_20 (Linux 4.4.0-128-generic)
CPU=ppc64le (4 logical CPUs) (0x1fe3a0000 RAM)
----------- Stack Backtrace -----------
(0x00003FFF8CF0CFE8 [libj9vm29.so+0xacfe8])
(0x00003FFF87D7E6BC [libj9jit29.so+0x95e6bc])
(0x00003FFF87D7F790 [libj9jit29.so+0x95f790])
(0x00003FFF8CF52918 [libj9vm29.so+0xf2918])
(0x00003FFF8CF0D6E4 [libj9vm29.so+0xad6e4])
(0x00003FFF8CF0E168 [libj9vm29.so+0xae168])
(0x00003FFF8CE79CF4 [libj9vm29.so+0x19cf4])
(0x00003FFF8CF403AC [libj9vm29.so+0xe03ac])
(0x00003FFF8CEF0BD4 [libj9vm29.so+0x90bd4])
(0x00003FFF8CF61D3C [libj9vm29.so+0x101d3c])
(0x00003FFF8CC9E89C [libj9prt29.so+0x2e89c])
(0x00003FFF8CF5D110 [libj9vm29.so+0xfd110])
(0x00003FFF8CE30B98 [libj9thr29.so+0x10b98])
(0x00003FFF8D958070 [libpthread.so.0+0x8070])
clone+0x98 (0x00003FFF8DB23A70 [libc.so.6+0x123a70])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2018/09/17 09:14:07 - please wait.
Unhandled exception
Type=Segmentation error vmState=0x00030000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00003FFF8CF0CA70 Handler2=00003FFF8CC9D2B0
R0=00003FFF87D7E6BC R1=00003FFF6D00B0B0 R2=00003FFF8D047000 R3=000000000040A600
R4=00003FFF6D00B388 R5=0000000000000000 R6=0000000000000000 R7=00003FFF6D00B140
R8=00003FFF6D00B148 R9=00003FFF8CF0CFE0 R10=0000000000000001 R11=00003FFF67755D7A
R12=00003FFF8CF0CFE0 R13=00003FFF6D016900 R14=0000000000409DE0 R15=000000000040A600
R16=00003FFF67710038 R17=FFFFFFFFFFFFFFFF R18=0000000000000080 R19=00003FFF6D3BF8B8
R20=0000000000000000 R21=0000000000014F00 R22=00000000000017A2 R23=0000000000000000
R24=00000000004109E8 R25=0000000000000000 R26=0000000000416180 R27=0000000000000000
R28=00003FFF6D00B490 R29=0000000000000000 R30=00003FFF6D00B4A8 R31=00003FFF6D00B388
NIP=00003FFF8CF0CFE8 MSR=800000010280F033 ORIG_GPR3=C000000000536370 CTR=00003FFF8CF0CFE0
LINK=00003FFF87D7E4CC XER=0000000020000000 CCR=0000000044004844 SOFTE=0000000000000001
TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
JVMDUMP032I JVM requested System dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/core.20180917.091407.27612.0001.dmp' in response to an event
FPR0 00000000003217d0 (f: 3282896.000000, d: 1.621966e-317)
FPR1 40527e7b5b081621 (f: 1527256576.000000, d: 7.397628e+01)
FPR2 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
FPR3 3c73e00000000000 (f: 0.000000, d: 1.723881e-17)
FPR4 402a56ef8ec92280 (f: 2395546112.000000, d: 1.316980e+01)
FPR5 3ff5fb9e00000000 (f: 0.000000, d: 1.373930e+00)
FPR6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR8 000000000040a50a (f: 4236554.000000, d: 2.093136e-317)
FPR9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR10 0061006500700065 (f: 7340133.000000, d: 7.565937e-307)
FPR11 37f0003a00000000 (f: 0.000000, d: 2.938898e-39)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 402af995e2afabd0 (f: 3803163648.000000, d: 1.348747e+01)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdkbinary/j2sdk-image/lib/compressedrefs/libj9vm29.so
Module_base_address=00003FFF8CE60000
Target=2_90_20180917_20 (Linux 4.4.0-128-generic)
CPU=ppc64le (4 logical CPUs) (0x1fe3a0000 RAM)
----------- Stack Backtrace -----------
(0x00003FFF8CF0CFE8 [libj9vm29.so+0xacfe8])
(0x00003FFF87D7E6BC [libj9jit29.so+0x95e6bc])
(0x00003FFF87D7F790 [libj9jit29.so+0x95f790])
(0x00003FFF8CF52918 [libj9vm29.so+0xf2918])
(0x00003FFF8CF0D6E4 [libj9vm29.so+0xad6e4])
(0x00003FFF8CF0E168 [libj9vm29.so+0xae168])
(0x00003FFF8CE79CF4 [libj9vm29.so+0x19cf4])
(0x00003FFF8CF403AC [libj9vm29.so+0xe03ac])
(0x00003FFF8CEF0BD4 [libj9vm29.so+0x90bd4])
(0x00003FFF8CF61D3C [libj9vm29.so+0x101d3c])
(0x00003FFF8CC9E89C [libj9prt29.so+0x2e89c])
(0x00003FFF8CF5D110 [libj9vm29.so+0xfd110])
(0x00003FFF8CE30B98 [libj9thr29.so+0x10b98])
(0x00003FFF8D958070 [libpthread.so.0+0x8070])
clone+0x98 (0x00003FFF8DB23A70 [libc.so.6+0x123a70])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2018/09/17 09:14:07 - please wait.
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.27640.
JVMDUMP010I System dump written to /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/core.20180917.091407.27612.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/javacore.20180917.091407.27612.0003.txt' in response to an event
JVMDUMP032I JVM requested System dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/core.20180917.091407.27612.0002.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.27644.
JVMDUMP010I System dump written to /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/core.20180917.091407.27612.0002.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/javacore.20180917.091407.27612.0004.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/javacore.20180917.091407.27612.0003.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/Snap.20180917.091407.27612.0005.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/Snap.20180917.091407.27612.0005.trc
JVMDUMP007I JVM Requesting JIT dump using '/home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/jitdump.20180917.091407.27612.0006.dmp'
JVMDUMP010I JIT dump written to /home/jenkins/jenkins-agent/workspace/Test-extended.functional-JDK11-linux_ppc-64_cmprssptrs_le/openjdk-tests/functional/Java8andUp/jitdump.20180917.091407.27612.0006.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
threadMXBeanTimersTest_3_FAILED
This failure looks different from the original reported failure. The first looked to be in jitted code, while this one looks to have crashed in the VM module. It's hard to say for sure if it's the same issue (though I wouldn't rule it out). If we still think this is the same issue, then it looks like this could be a common problem across all platforms.
@andrewcraik , @zl-wang - either ppc or common issue... can we have someone look at it.
@IBMJimmyk - would you please look at this.
I was able to reproduce the crash in my personal workspace. Failure rate seems to be around 1 in 20 but the test is fairly quick to run. However, I am having problem with my test environment that prevents the generation of a corefile so I still need to get that sorted out.
I got a corefile. It looks like this one was crashing inside growSlotIterator.
0x000072120e08cfe8 {libj9vm29.so}{growSlotIterator} [0x7211d3fab000] 0x000072120d5ae7f0 {libj9jit29.so}{jitWalkStackFrames} [0x7211d3fab110] 0x000072120e0d2918 {libj9vm29.so}{walkStackFrames} [0x7211d3fab190] 0x000072120e08d6e4 {libj9vm29.so}{internalGrowJavaStack} [0x7211d3fab260] 0x000072120e08e168 {libj9vm29.so}{growJavaStack} [0x7211d3fab5e0] 0x000072120dff9cf4 {libj9vm29.so}{bytecodeLoop} [0x7211d3fab610] 0x000072120e0c03ac {libj9vm29.so}{c_cInterpreter} [0x7211d3faba80]
Crash occurs just after entering growSlotIterator. It tries to load from $r5 but $r5 is 0x0.
This is likely a duplicate. See this comment: https://github.com/eclipse/openj9/issues/2917#issuecomment-422992665
This test has a low failure rate (maybe around 1 in 20). I'll need to run it a bunch of times to make sure I fixed the problem.
Java Version:
Output:
Link: https://ci.adoptopenjdk.net/view/work%20in%20progress/job/TestBuild_Sandbox/492/