search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JDK8 zLInux Illegal instruction vmState=0x00000000 in jitFillOSRBuffer () #9514

Closed JasonFengJ9 closed 1 year ago

JasonFengJ9 commented 4 years ago

Failure link

From an internal build OpenJ9 acceptance build 446441 - job 642439837

p> 20200510-10:01:41 java version "1.8.0_251"
p> 20200510-10:01:41 Java(TM) SE Runtime Environment (build 8.0.7.0 - pxz6480sr7-20200428_02(SR7))
p> 20200510-10:01:41 IBM J9 VM (build 2.9, JRE 1.8.0 Linux s390x-64-Bit 20200510_446441 (JIT enabled, AOT enabled)
p> 20200510-10:01:41 OpenJ9   - 0712ae8
p> 20200510-10:01:41 OMR      - 295075e
p> 20200510-10:01:41 IBM      - ddee647)
p> 20200510-10:01:41 JCL - 20200407_01 based on Oracle jdk8u251-b08

Optional info

Failure output (captured from console output)

p> 20200510-10:01:41 java version "1.8.0_251"
p> 20200510-10:01:41 Java(TM) SE Runtime Environment (build 8.0.7.0 - pxz6480sr7-20200428_02(SR7))
p> 20200510-10:01:41 IBM J9 VM (build 2.9, JRE 1.8.0 Linux s390x-64-Bit 20200510_446441 (JIT enabled, AOT enabled)
p> 20200510-10:01:41 OpenJ9   - 0712ae8
p> 20200510-10:01:41 OMR      - 295075e
p> 20200510-10:01:41 IBM      - ddee647)
p> 20200510-10:01:41 JCL - 20200407_01 based on Oracle jdk8u251-b08

j> 10:02:09 
j> 10:02:09 *** Invalid JIT return address 6966696573207468 in 000003FF6E4F1D10
j> 10:02:09 
j> 10:02:09 
j> 10:02:09 
j> 10:02:09 *** Invalid JIT return address 0000000000000000 in 000003FF6E4F1D10
j> 10:02:09 

p> 20200510-10:02:58 <gdb> #12 <signal handler called>
p> 20200510-10:02:58 <gdb> #13 0x000003ff75150324 in ?? ()
p> 20200510-10:02:58 <gdb> #14 0x000003ffb1fa571c in jitFillOSRBuffer () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9vm29.so
p> 20200510-10:02:58 <gdb> #15 0x000003ffb16d385a in performOSR () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so
p> 20200510-10:02:58 <gdb> #16 0x000003ffb16dabc6 in induceOSROnCurrentThread () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so
p> 20200510-10:02:58 <gdb> #17 0x000003ffb16d1506 in old_slow_jitInduceOSRAtCurrentPC () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so
p> 20200510-10:02:58 <gdb> #18 0x000003ffb16e3c5e in jitInduceOSRAtCurrentPC () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so
p> 20200510-10:02:58 <gdb> #19 0x000003ffb1f9fe88 in runJavaThread () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9vm29.so
p> 20200510-10:02:58 <gdb> #20 0x000003ffb1ffb0fe in javaProtectedThreadProc () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9vm29.so
p> 20200510-10:02:58 <gdb> #21 0x000003ffb1da24e0 in omrsig_protect () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9prt29.so
p> 20200510-10:02:58 <gdb> #22 0x000003ffb1ff7254 in javaThreadProc () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9vm29.so
p> 20200510-10:02:58 <gdb> #23 0x000003ffb1e85958 in thread_wrapper () from /j9vm/ascii/builds/bld_446441/sdk/xz6480/jre/lib/s390x/default/libj9thr29.so
p> 20200510-10:02:58 <gdb> #24 0x0000004d3a546556 in start_thread () from /lib64/libpthread.so.0
p> 20200510-10:02:58 <gdb> #25 0x0000004d3a3df8a6 in thread_start () from /lib64/libc.so.6
p> 20200510-10:02:58

There is another similar failure at job 642440153:

Attempting to redefine instrumented classs with new method added...
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=7C91A5B0 Handler2=7CB624B8
gpr0=00000016 gpr1=10056488 gpr2=10577210 gpr3=00000000
gpr4=7A7E1BDC gpr5=77980838 gpr6=00000001 gpr7=00000000
gpr8=00000000 gpr9=00000000 gpr10=7CA0AD00 gpr11=7D5E5F70
gpr12=7B654000 gpr13=7CA5F400 gpr14=FB62DD02 gpr15=7D5E5D68
hgpr0=00000000 hgpr1=77980838 hgpr2=781D6950 hgpr3=00000000
hgpr4=00000000 hgpr5=00000000 hgpr6=00000076 hgpr7=00000200
hgpr8=00000040 hgpr9=7CAEBF00 hgpr10=7C547000 hgpr11=7C2CAD3C
hgpr12=FC5262EA hgpr13=00000000 hgpr14=00000000 hgpr15=00000000
psw=FA7E1BF8 mask=070DD000 fpc=00080000 bea=77980838
fpr0 42a014dd7c9d9424 (f: 2090701824.000000, d: 8.840901e+12)
fpr1 bd189a287c9d9424 (f: 2090701824.000000, d: -2.185113e-14)
fpr2 3e3ab283000000dc (f: 220.000000, d: 6.215952e-09)
fpr3 3d1aac007c9d9424 (f: 2090701824.000000, d: 2.368939e-14)
fpr4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr5 33fd3be07c9d9424 (f: 2090701824.000000, d: 2.910768e-58)
fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr7 3c97ce3f00000000 (f: 0.000000, d: 8.259244e-17)
fpr8 39b409e900000000 (f: 0.000000, d: 9.879848e-31)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 3c07528400000000 (f: 0.000000, d: 1.580382e-19)
fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr12 388f3d0900000000 (f: 0.000000, d: 2.937647e-36)
fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr14 3e17cee700000000 (f: 0.000000, d: 1.385820e-09)
fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=sun/instrument/InstrumentationImpl.redefineClasses([Ljava/lang/instrument/ClassDefinition;)V
Target=2_90_20200510_446441 (Linux 2.6.32-754.23.1.el6.s390x)
CPU=s390 (4 logical CPUs) (0xf34e0000 RAM)
----------- Stack Backtrace -----------
(0x7CB80270 [libj9prt29.so+0x37270])
(0x7CB637E6 [libj9prt29.so+0x1a7e6])
(0x7CB80304 [libj9prt29.so+0x37304])
(0x7CB803FA [libj9prt29.so+0x373fa])
(0x7CB800C2 [libj9prt29.so+0x370c2])
(0x7CB637E6 [libj9prt29.so+0x1a7e6])
(0x7CB8015A [libj9prt29.so+0x3715a])
(0x7C91A510 [libj9vm29.so+0x91510])
(0x7CB637E6 [libj9prt29.so+0x1a7e6])
(0x7C91A6CC [libj9vm29.so+0x916cc])
(0x7CB62712 [libj9prt29.so+0x19712])
(0x7D5E5B70 [<unknown>+0x0])
(0x7A7E1BF8 [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2020/05/10 10:00:32 - please wait.
JVMDUMP032I JVM requested System dump using '/tmp/bld_446441/javaagentASMBCITest/642440153/core.20200510.100032.7542.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.7560.

JVMDUMP010I System dump written to /tmp/bld_446441/javaagentASMBCITest/642440153/core.20200510.100032.7542.0001.dmp
JVMDUMP032I JVM requested Java dump using '/tmp/bld_446441/javaagentASMBCITest/642440153/javacore.20200510.100032.7542.0002.txt' in response to an event

*** Invalid JIT return address 726F7065 in 7D5E1060

These two failures are intermittent, first one wasn't reproduced in a 10x grinder, the second one failed 1/10 in another 10x grinder.

JasonFengJ9 commented 1 year ago

JDK8 zLinux VM_Sanity.TestRefreshGCSpecialClassesCache_BCI_EXTENDED_HCR.Mode107.1(lnxec418)

java version "1.8.0_381"
Java(TM) SE Runtime Environment (build 8.0.9.0 - pxz6480sr9-20230615_01(SR9))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux s390x-64-Bit 20230705_53942 (JIT enabled, AOT enabled)
OpenJ9   - 3bf9cb0
OMR      - 357a43d
IBM      - 696e9df)
JCL - 20230608_01 based on Oracle jdk8u381-b06

starting javaagentASMBCITest
Inside Java agent! 
Suite run count = 1
Starting test_EXTENDED_HCR(com.ibm.javaagent.junit.RefreshGCCache_ExtendedHCR_Test)
Attempting BCI...
Instrumenting java/lang/ClassLoader by adding a new method.
Attempting to add call in method : getResource
Attempting to add call in method : getResources
Attempting to add call in method : getResourceAsStream
BCI complete
Attempting to redefine instrumented classs with new method added...
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF98666080 Handler2=000003FF983ACD38
gpr0=0000000000000016 gpr1=000003FF9907B838 gpr2=000000000000000E gpr3=000003FF98738060
gpr4=000003FF5BF81654 gpr5=000003FF947D0830 gpr6=0000000000000000 gpr7=0000000000002502
gpr8=000003FF8D8A2B68 gpr9=000003FF9408B330 gpr10=0000000000000000 gpr11=000003FF8D634BC8
gpr12=000003FF8DA14174 gpr13=000003FF94087900 gpr14=000003FF94085030 gpr15=000003FF9907B8C0
psw=000003FF5BF8165C mask=0705100180000000 fpc=0008fe00 bea=000003FF986A7908
fpr0 42275d2f00000000 (f: 0.000000, d: 5.017380e+10)
fpr1 3e638e29ac782e4f (f: 2893557248.000000, d: 3.642463e-08)
fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
fpr3 367f00c000000000 (f: 0.000000, d: 3.394090e-46)
fpr4 3b7fd580944ad818 (f: 2487932928.000000, d: 4.213194e-22)
fpr5 3e924925a000c000 (f: 2684403712.000000, d: 2.724786e-07)
fpr6 3ecccccd9907cb98 (f: 2567425024.000000, d: 3.433229e-06)
fpr7 3e3a332580008000 (f: 2147516416.000000, d: 6.100114e-09)
fpr8 000002aa21766070 (f: 561406080.000000, d: 1.447479e-311)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=sun/instrument/InstrumentationImpl.redefineClasses([Ljava/lang/instrument/ClassDefinition;)V
Target=2_90_20230705_53942 (Linux 3.10.0-1160.88.1.el7.s390x)
CPU=s390x (2 logical CPUs) (0xfa8b9000 RAM)
----------- Stack Backtrace -----------
 (0x000003FF5BF8165C [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/07/05 13:16:21 - please wait.
JVMDUMP032I JVM requested System dump using '/tmp/bld_53942/javaagentASMBCITest/63387562/core.20230705.131621.22532.0001.dmp' in response to an event
JVMDUMP010I System dump written to /tmp/bld_53942/javaagentASMBCITest/63387562/core.20230705.131621.22532.0001.dmp
JVMDUMP032I JVM requested Java dump using '/tmp/bld_53942/javaagentASMBCITest/63387562/javacore.20230705.131621.22532.0002.txt' in response to an event

*** Invalid JIT return address 0000000000000031 in 000003FF99075740

100x grinder - 2/100 failed

JasonFengJ9 commented 1 year ago

JDK8 Linux S390 64bit VM_Sanity.TestRefreshGCSpecialClassesCache_BCI_EXTENDED_HCR.Mode107.1(lnxec418)

java version "1.8.0_381"
Java(TM) SE Runtime Environment (build 8.0.9.0 - pxz6480sr9-20230615_01(SR9))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux s390x-64-Bit 20230705_53942 (JIT enabled, AOT enabled)
OpenJ9   - 3bf9cb0
OMR      - 357a43d
IBM      - 696e9df)
JCL - 20230608_01 based on Oracle jdk8u381-b06

starting javaagentASMBCITest
Inside Java agent! 
Suite run count = 1
Starting test_EXTENDED_HCR(com.ibm.javaagent.junit.RefreshGCCache_ExtendedHCR_Test)
Attempting BCI...
Instrumenting java/lang/ClassLoader by adding a new method.
Attempting to add call in method : getResource
Attempting to add call in method : getResources
Attempting to add call in method : getResourceAsStream
BCI complete
Attempting to redefine instrumented classs with new method added...
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF98666080 Handler2=000003FF983ACD38
gpr0=0000000000000016 gpr1=000003FF9907B838 gpr2=000000000000000E gpr3=000003FF98738060
gpr4=000003FF5BF81654 gpr5=000003FF947D0830 gpr6=0000000000000000 gpr7=0000000000002502
gpr8=000003FF8D8A2B68 gpr9=000003FF9408B330 gpr10=0000000000000000 gpr11=000003FF8D634BC8
gpr12=000003FF8DA14174 gpr13=000003FF94087900 gpr14=000003FF94085030 gpr15=000003FF9907B8C0
psw=000003FF5BF8165C mask=0705100180000000 fpc=0008fe00 bea=000003FF986A7908
fpr0 42275d2f00000000 (f: 0.000000, d: 5.017380e+10)
fpr1 3e638e29ac782e4f (f: 2893557248.000000, d: 3.642463e-08)
fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
fpr3 367f00c000000000 (f: 0.000000, d: 3.394090e-46)
fpr4 3b7fd580944ad818 (f: 2487932928.000000, d: 4.213194e-22)
fpr5 3e924925a000c000 (f: 2684403712.000000, d: 2.724786e-07)
fpr6 3ecccccd9907cb98 (f: 2567425024.000000, d: 3.433229e-06)
fpr7 3e3a332580008000 (f: 2147516416.000000, d: 6.100114e-09)
fpr8 000002aa21766070 (f: 561406080.000000, d: 1.447479e-311)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=sun/instrument/InstrumentationImpl.redefineClasses([Ljava/lang/instrument/ClassDefinition;)V
Target=2_90_20230705_53942 (Linux 3.10.0-1160.88.1.el7.s390x)
CPU=s390x (2 logical CPUs) (0xfa8b9000 RAM)
----------- Stack Backtrace -----------
 (0x000003FF5BF8165C [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/07/05 13:16:21 - please wait.
JVMDUMP032I JVM requested System dump using '/tmp/bld_53942/javaagentASMBCITest/63387562/core.20230705.131621.22532.0001.dmp' in response to an event
JVMDUMP010I System dump written to /tmp/bld_53942/javaagentASMBCITest/63387562/core.20230705.131621.22532.0001.dmp
JVMDUMP032I JVM requested Java dump using '/tmp/bld_53942/javaagentASMBCITest/63387562/javacore.20230705.131621.22532.0002.txt' in response to an event

*** Invalid JIT return address 0000000000000031 in 000003FF99075740

100x grinder - 2/100 failed

pshipton commented 1 year ago

@r30shah Pls take a look at these recent failures. They may only happen on certain machines (lnxec418, lnxec705, lnxec706) and I've disabled these machines now. Possibly they were disabled in the past but were re-enabled due the recent maintenance.

r30shah commented 1 year ago

Taking a look, I no longer have access to these images (something changed), so requested access, Will investigate once I have access.

r30shah commented 1 year ago

@pshipton You are correct this is the same issue as https://github.com/eclipse-openj9/openj9/issues/9876, we can close this one as duplicate of that one. @VermaSh will check out if we can resolve this one or not.