search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JIT crashed during compilation #2699

Closed d0k1 closed 6 years ago

d0k1 commented 6 years ago

Got OpenJ9 crash during a webapp startup. OpenJ9 created some disgnostic info: snapdump, javadump, jitdump, coredump Javadump

0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TICHARSET     UTF-8
1TISIGINFO     Dump Event "gpf" (00002000) received
1TIDATETIME    Date: 2018/08/24 at 14:01:30:099
1TIREQFLAGS    Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE   Prep State: 0x100 (trace_disabled)
1TIPREPINFO    Exclusive VM access not taken: data may not be consistent across javacore sections
NULL           ------------------------------------------------------------------------
0SECTION       GPINFO subcomponent dump routine
NULL           ================================
2XHOSLEVEL     OS Level         : Linux 3.10.0-693.21.1.el7.x86_64
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : amd64
3XHNUMCPUS       How Many       : 8
3XHNUMASUP       NUMA is either not supported or has been disabled by user
NULL
1XHEXCPCODE    J9Generic_Signal_Number: 00000004
1XHEXCPCODE    Signal_Number: 0000000B
1XHEXCPCODE    Error_Value: 00000000
1XHEXCPCODE    Signal_Code: 00000001
1XHEXCPCODE    Handler1: 00007FEA09A2A770
1XHEXCPCODE    Handler2: 00007FEA08E215D0
1XHEXCPCODE    InaccessibleAddress: 0000000000000030
NULL
1XHEXCPMODULE  Module: /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
1XHEXCPMODULE  Module_base_address: 00007FE9FCF0B000
NULL
1XHREGISTERS   Registers:
....
1XHEXCPMODULE  Compiling method: org/hibernate/engine/internal/TwoPhaseLoad.doInitializeEntity(Ljava/lang/Object;Lorg/hibernate/engine/spi/EntityEntry;ZLorg/hibernate/engine/s
pi/SessionImplementor;Lorg/hibernate/event/spi/PreLoadEvent;)V
NULL
1XHFLAGS       VM flags:00000000000501FF
NULL
NULL           ------------------------------------------------------------------------
0SECTION       ENVINFO subcomponent dump routine
NULL           =================================
1CIJAVAVERSION JRE 1.8.0 Linux amd64-64 (build 1.8.0_181-internal-_2018_08_19_15_51-b00)
1CIVMVERSION   20180819_000000
1CIJ9VMVERSION 99cb580
1CIJITVERSION  99cb580
1CIOMRVERSION  f71c469_CMPRSS
1CIJCLVERSION  2b004fd based on jdk8u181-b13
1CIJITMODES    JIT enabled, AOT enabled, FSD enabled, HCR enabled
1CIRUNNINGAS   Running as a standalone JVM
1CIVMIDLESTATE VM Idle State: ACTIVE
1CISTARTTIME   JVM start time: 2018/08/24 at 13:53:24:532
1CISTARTNANO   JVM start nanotime: 6753879174526618
1CIPROCESSID   Process ID: 24959 (0x617F)
0SECTION       THREADS subcomponent dump routine
NULL           =================================
NULL
1XMPOOLINFO    JVM Thread pool info:
2XMPOOLTOTAL       Current total number of pooled threads: 127
2XMPOOLLIVE        Current total number of live threads: 121
2XMPOOLDAEMON      Current total number of live daemon threads: 72
NULL
1XMCURTHDINFO  Current thread
3XMTHREADINFO      "JIT Compilation Thread-2" J9VMThread:0x000000000120DD00, omrthread_t:0x00007FEA040C4DA0, java/lang/Thread:0x000000048075AA80, state:R, prio=10
3XMJAVALTHREAD            (java/lang/Thread getId:0xB, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x6184, native priority:0xB, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)
3XMTHREADINFO2            (native stack address range from:0x00007FE9D8DA7000, to:0x00007FE9D8EA8000, size:0x101000)
3XMCPUTIME               CPU usage total: 106.916800334 secs, current category="JIT"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           No Java callstack associated with this thread
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007FEA08E3D2D2 [libj9prt29.so+0x3b2d2])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08E3D34C [libj9prt29.so+0x3b34c])
4XENATIVESTACK               (0x00007FEA08E3D456 [libj9prt29.so+0x3b456])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08E3D1A1 [libj9prt29.so+0x3b1a1])
4XENATIVESTACK               (0x00007FEA08E392EC [libj9prt29.so+0x372ec])
4XENATIVESTACK               (0x00007FEA08E3A4A4 [libj9prt29.so+0x384a4])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08BD9DA8 [libj9dmp29.so+0x18da8])
4XENATIVESTACK               (0x00007FEA08BDA42D [libj9dmp29.so+0x1942d])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08BD6DCF [libj9dmp29.so+0x15dcf])
4XENATIVESTACK               (0x00007FEA08BD281D [libj9dmp29.so+0x1181d])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08BD35F7 [libj9dmp29.so+0x125f7])
4XENATIVESTACK               (0x00007FEA08BDC559 [libj9dmp29.so+0x1b559])
4XENATIVESTACK               (0x00007FEA08BC5DDD [libj9dmp29.so+0x4ddd])
4XENATIVESTACK               (0x00007FEA08BC53C5 [libj9dmp29.so+0x43c5])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA08BC8CBA [libj9dmp29.so+0x7cba])
4XENATIVESTACK               (0x00007FEA08BC8E27 [libj9dmp29.so+0x7e27])
4XENATIVESTACK               (0x00007FEA08BDE28B [libj9dmp29.so+0x1d28b])
4XENATIVESTACK               (0x00007FEA09A2A34E [libj9vm29.so+0x9a34e])
4XENATIVESTACK               (0x00007FEA08E2219D [libj9prt29.so+0x2019d])
4XENATIVESTACK               (0x00007FEA09A2A4F0 [libj9vm29.so+0x9a4f0])
4XENATIVESTACK               (0x00007FEA08E21829 [libj9prt29.so+0x1f829])
4XENATIVESTACK               (0x00007FEA0ABA95E0 [libpthread.so.0+0xf5e0])
4XENATIVESTACK               (0x00007FE9FD09294F [libj9jit29.so+0x18794f])
4XENATIVESTACK               (0x00007FE9FD093845 [libj9jit29.so+0x188845])
4XENATIVESTACK               (0x00007FE9FD0A13FC [libj9jit29.so+0x1963fc])
4XENATIVESTACK               (0x00007FE9FD0B19DF [libj9jit29.so+0x1a69df])

gdb backtrace 

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fea0aba69b1 in ?? ()
(gdb) bt
#0  0x00007fea0aba69b1 in ?? ()
#1  0x00007fea08e3dae5 in omrdump_create () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#2  0x00007fea08bc998d in doSystemDump () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9dmp29.so
#3  0x00007fea08bc53c5 in protectedDumpFunction () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9dmp29.so
#4  0x00007fea08e2219d in omrsig_protect () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#5  0x00007fea08bc8cba in runDumpFunction () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9dmp29.so
#6  0x00007fea08bc8e27 in runDumpAgent () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9dmp29.so
#7  0x00007fea08bde28b in triggerDumpAgents () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9dmp29.so
#8  0x00007fea09a2a34e in generateDiagnosticFiles () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9vm29.so
#9  0x00007fea08e2219d in omrsig_protect () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#10 0x00007fea09a2a4f0 in vmSignalHandler () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9vm29.so
#11 0x00007fea08e21829 in masterSynchSignalHandler () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#12 <signal handler called>
#13 0x00007fe9fd09294f in TR_ClassLookahead::examineNode(TR::TreeTop*, TR::Node*, TR::Node*, int, TR::Node*, unsigned short) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#14 0x00007fe9fd093845 in TR_ClassLookahead::perform() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#15 0x00007fe9fd0a13fc in TR_J9ByteCodeIlGenerator::performClassLookahead(TR_PersistentClassInfo*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#16 0x00007fe9fd0b19df in TR_J9ByteCodeIlGenerator::genInvoke(TR::SymbolReference*, TR::Node*, TR::Node*) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#17 0x00007fe9fd0b683f in TR_J9ByteCodeIlGenerator::walker(TR::Block*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#18 0x00007fe9fd09956e in TR_J9ByteCodeIlGenerator::genILFromByteCodes() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#19 0x00007fe9fd09be78 in TR_J9ByteCodeIlGenerator::genIL() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#20 0x00007fe9fd31a3ec in OMR::ResolvedMethodSymbol::genIL(TR_FrontEnd*, TR::Compilation*, TR::SymbolReferenceTable*, TR::IlGenRequest&) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#21 0x00007fe9fd0c7b01 in TR_J9InlinerPolicy::_tryToGenerateILForMethod(TR::ResolvedMethodSymbol*, TR::ResolvedMethodSymbol*, TR_CallTarget*) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#22 0x00007fe9fd3ce738 in TR_InlinerBase::inlineCallTarget2(TR_CallStack*, TR_CallTarget*, TR::TreeTop**, bool, int) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#23 0x00007fe9fd0e29b9 in TR_InlinerBase::inlineCallTarget(TR_CallStack*, TR_CallTarget*, bool, TR_PrexArgInfo*, TR::TreeTop**) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#24 0x00007fe9fd3ce37e in TR_InlinerBase::inlineFromGraph(TR_CallStack*, TR_CallTarget*, TR_InnerPreexistenceInfo*) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#25 0x00007fe9fd3cf0a1 in TR_InlinerBase::inlineCallTarget2(TR_CallStack*, TR_CallTarget*, TR::TreeTop**, bool, int) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#26 0x00007fe9fd0e29b9 in TR_InlinerBase::inlineCallTarget(TR_CallStack*, TR_CallTarget*, bool, TR_PrexArgInfo*, TR::TreeTop**) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#27 0x00007fe9fd0d093b in TR_MultipleCallTargetInliner::inlineCallTargets(TR::ResolvedMethodSymbol*, TR_CallStack*, TR_InnerPreexistenceInfo*) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#28 0x00007fe9fd3d1003 in TR_InlinerBase::performInlining(TR::ResolvedMethodSymbol*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#29 0x00007fe9fd0c9342 in TR_Inliner::perform() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#30 0x00007fe9fd476b33 in OMR::Optimizer::performOptimization(OptimizationStrategy const*, int, int, int) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#31 0x00007fe9fd476292 in OMR::Optimizer::performOptimization(OptimizationStrategy const*, int, int, int) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#32 0x00007fe9fd477c42 in OMR::Optimizer::optimize() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#33 0x00007fe9fd2cc09e in OMR::Compilation::performOptimizations() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#34 0x00007fe9fd2d153e in OMR::Compilation::compile() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#35 0x00007fe9fd03729a in TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, TR_J9VMBase&, TR_OptimizationPlan*, TR::SegmentAllocator const&) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
---Type <return> to continue, or q <return> to quit---
#36 0x00007fe9fd03828e in TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*, void*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#37 0x00007fea08e2219d in omrsig_protect () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#38 0x00007fe9fd039b51 in TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR_MethodToBeCompiled*, J9::J9SegmentProvider&) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#39 0x00007fe9fd039ff1 in TR::CompilationInfoPerThread::processEntry(TR_MethodToBeCompiled&, J9::J9SegmentProvider&) ()
   from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#40 0x00007fe9fd03a640 in TR::CompilationInfoPerThread::processEntries() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#41 0x00007fe9fd03a90a in TR::CompilationInfoPerThread::run() () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#42 0x00007fe9fd03a9bf in protectedCompilationThreadProc(J9PortLibrary*, TR::CompilationInfoPerThread*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#43 0x00007fea08e2219d in omrsig_protect () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9prt29.so
#44 0x00007fe9fd03ad06 in compilationThreadProc(void*) () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9jit29.so
#45 0x00007fea09783037 in thread_wrapper () from /opt/openj9_8u_181/jre/lib/amd64/compressedrefs/libj9thr29.so
#46 0x00007fea0aba1e25 in ?? ()
#47 0x0000000000000000 in ?? ()
fjeremic commented 6 years ago

How reproducible is the failure? Is there a way for us to reproduce the issue? @cathyzhyi FYI crash during IL gen. which we may need to look at.

If the issue is reproducible and it occurs in the same method (you can find the name in the above javacore by searching for "Compiling method:") then please grab a trace log of the failure by adding the following to your commandline:

-Xjit:{org/hibernate/engine/internal/TwoPhaseLoad.doInitializeEntity(Ljava/lang/Object;Lorg/hibernate/engine/spi/EntityEntry;ZLorg/hibernate/engine/spi/SessionImplementor;Lorg/hibernate/event/spi/PreLoadEvent;)V}(tracefull,traceILGen,log=doInitializeEntity.trace)

It will generate a file in the pwd called doInitializeEntity.trace. Please upload this file and the associated javacore, and coredump to somewhere we can access them and we can take a look.

DanHeidinga commented 6 years ago

@d0k1 Have you been able to reproduce this with the logging suggested by @fjeremic?

d0k1 commented 6 years ago

@DanHeidinga I couldn't reproduce it, sorry. But now I know what debug info is needed to investigate a problem.

DanHeidinga commented 6 years ago

Thanks @d0k1 - hopefully that's info you won't need in the future ;)

allenliu88 commented 9 months ago

@DanHeidinga I couldn't reproduce it, sorry. But now I know what debug info is needed to investigate a problem.

Finally how did you solve it? we currently encounter the exactly the same error, like this:

1XHEXCPCODE J9Generic_Signal_Number: 00000004 1XHEXCPCODE Signal_Number: 0000000B 1XHEXCPCODE Error_Value: 00000000 1XHEXCPCODE Signal_Code: 00000080 1XHEXCPCODE Handler1: 00007F6006F35890 1XHEXCPCODE Handler2: 00007F6006833450 1XHEXCPCODE InaccessibleAddress: 0000000000000000 NULL
1XHEXCPMODULE Module: /opt/java/openjdk/jre/lib/amd64/compressedrefs/libj9jit29.so 1XHEXCPMODULE Module_base_address: 00007F60049ED000 ... 1XHEXCPMODULE Compiling method: io/micrometer/core/instrument/Meter$Id.hashCode()I NULL
1XHFLAGS VM flags:00000000000501FF

pshipton commented 9 months ago

@allenliu88 pls open a new issue and provide all the details. If you aren't using the latest release, pls try the latest.