Closed ehrenjulzert closed 2 years ago
I see a slightly different stack trace re-running the above test:
#12 <signal handler called>
#13 walkBytecodeFrame (walkState=0x7fedf5683730) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/swalk.c:972
#14 walkStackFrames (currentThread=<optimized out>, walkState=0x7fedf5683730) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/swalk.c:313
#15 0x00007fedf47907b6 in GC_VMThreadStackSlotIterator::scanSlots (vmThread=<optimized out>, walkThread=walkThread@entry=0x1aa00, userData=userData@entry=0x7fedf5683a40,
oSlotIterator=oSlotIterator@entry=0x7fedf4788c50 <stackSlotIterator(J9JavaVM*, J9Object**, void*, J9StackWalkState*, void const*)>, includeStackFrameClassReferences=<optimized out>,
trackVisibleFrameDepth=<optimized out>) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_structs/VMThreadStackSlotIterator.cpp:114
#16 0x00007fedf478875d in MM_RootScanner::scanOneThread (this=0x7fedf5683b30, env=0x7fedf0048ea8, walkThread=0x1aa00, localData=0x7fedf5683a40)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_base/RootScanner.cpp:524
#17 0x00007fedf478746f in MM_RootScanner::scanThreads (this=0x7fedf5683b30, env=0x7fedf0048ea8) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_base/RootScanner.cpp:493
#18 0x00007fedf478a032 in MM_RootScanner::scanRoots (this=0x7fedf5683b30, env=0x7fedf0048ea8) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_base/RootScanner.cpp:924
#19 0x00007fedf48d27eb in MM_ScavengerRootScanner::scanRoots (env=0x7fedf0048ea8, this=0x7fedf5683b30)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:200
#20 MM_Scavenger::workThreadGarbageCollect (this=0x7fedf0080fd0, env=0x7fedf0048ea8) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/standard/Scavenger.cpp:2582
#21 0x00007fedf487e1ce in MM_ParallelDispatcher::run (this=0x7fedf0046d10, env=0x7fedf0048ea8, task=0x7fedf5683c70, newThreadCount=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/ParallelDispatcher.cpp:588
#22 0x00007fedf48bcc0d in MM_Scavenger::scavenge (this=0x7fedf0080fd0, envBase=0x7fedf0048ea8) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/standard/Scavenger.cpp:580
#23 0x00007fedf48cad95 in MM_Scavenger::mainThreadGarbageCollect (this=0x7fedf0080fd0, envBase=0x7fedf0048ea8, allocDescription=<optimized out>, initMarkMap=<optimized out>, rebuildMarkBits=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/standard/Scavenger.cpp:4223
#24 0x00007fedf48cbed0 in MM_Scavenger::internalGarbageCollect (this=0x7fedf0080fd0, envBase=0x7fedf0048ea8, subSpace=0x7fedf008ce90, allocDescription=0x7fedf5684148)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/standard/Scavenger.cpp:4753
#25 0x00007fedf485b5d6 in MM_Collector::garbageCollect (this=0x7fedf0080fd0, env=0x7fedf0048ea8, callingSubSpace=0x7fedf008ce90, allocateDescription=0x7fedf5684148, gcCode=<optimized out>,
objectAllocationInterface=0x7fedf00bf780, baseSubSpace=0x7fedf008ce90, context=0x0) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/Collector.cpp:500
#26 0x00007fedf4928b2d in MM_MemorySubSpaceSemiSpace::allocationRequestFailed (this=0x7fedf008ce90, env=0x7fedf0048ea8, allocateDescription=0x7fedf5684148,
allocationType=MM_MemorySubSpace::ALLOCATION_TYPE_TLH, objectAllocationInterface=0x7fedf00bf780, baseSubSpace=<optimized out>, previousSubSpace=0x7fedf008caf0)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/MemorySubSpaceSemiSpace.cpp:137
#27 0x00007fedf490f1b8 in MM_MemorySubSpaceGeneric::allocateTLH (this=0x7fedf008caf0, env=0x7fedf0048ea8, allocDescription=0x7fedf5684148, objectAllocationInterface=0x7fedf00bf780, baseSubSpace=0x0,
previousSubSpace=<optimized out>, shouldCollectOnFailure=true) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/MemorySubSpaceGeneric.cpp:377
#28 0x00007fedf48852d5 in MM_TLHAllocationSupport::refresh (this=this@entry=0x7fedf00bf828, env=0x7fedf0048ea8, allocDescription=allocDescription@entry=0x7fedf5684148, shouldCollectOnFailure=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/TLHAllocationSupport.cpp:239
#29 0x00007fedf48854a6 in MM_TLHAllocationSupport::allocateFromTLH (this=0x7fedf00bf828, env=<optimized out>, allocDescription=0x7fedf5684148, shouldCollectOnFailure=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/TLHAllocationSupport.cpp:310
#30 0x00007fedf48845cf in MM_TLHAllocationInterface::allocateObject (this=0x7fedf00bf780, env=0x7fedf0048ea8, allocDescription=0x7fedf5684148, memorySpace=0x7fedf0092810, shouldCollectOnFailure=true)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/TLHAllocationInterface.cpp:194
#31 0x00007fedf488a0fb in MM_AllocateInitialization::allocateAndInitializeObject (omrVMThread=<optimized out>, this=0x7fedf5684130)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/base/AllocateInitialization.hpp:201
#32 OMR_GC_AllocateObject (omrVMThread=<optimized out>, allocator=allocator@entry=0x7fedf5684130) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/gc/startup/omrgcalloc.cpp:39
--Type <RET> for more, q to quit, c to continue without paging--
#33 0x00007fedf4796291 in J9AllocateObject (vmThread=0x1aa00, clazz=0x1d6300, allocateFlags=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/gc_modron_startup/mgcalloc.cpp:414
#34 0x00007fedf5482de8 in VM_ValueTypeHelpersCompressed::getFlattenedFieldAtOffset (fastPath=<optimized out>, srcOffset=<optimized out>, srcObject=<optimized out>, returnObjectClass=0x1d6300,
objectAllocate=..., objectAccessBarrier=..., currentThread=<optimized out>) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/ValueTypeHelpers.hpp:350
#35 VM_BytecodeInterpreterCompressed::inlUnsafeGetValue (_pc=<optimized out>, _sp=<optimized out>, this=<optimized out>)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/BytecodeInterpreter.hpp:4029
#36 VM_BytecodeInterpreterCompressed::run (this=0x7fedf5684800, vmThread=0x0) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/BytecodeInterpreter.hpp:10352
#37 0x00007fedf546a5a5 in bytecodeLoopCompressed (currentThread=<optimized out>) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/BytecodeInterpreter.inc:112
#38 0x00007fedf5523242 in c_cInterpreter () at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/build/linux-x86_64-server-release/vm/runtime/vm/xcinterp.s:158
#39 0x00007fedf53f38ef in runCallInMethod (env=0x1126b2, receiver=0x0, clazz=0x119190, methodID=0x7fedf0616d38, args=0x7fedf5684d78)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/callin.cpp:1132
#40 0x00007fedf5417919 in gpProtectedRunCallInMethod (entryArg=0x7fedf5684d30) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/jnicsup.cpp:301
#41 0x00007fedf520fc43 in omrsig_protect (portLibrary=0x7fedf5641380 <j9portLibrary>, fn=0x7fedf552e640 <signalProtectAndRunGlue>, fn_arg=0x7fedf5684cd0, handler=0x7fedf5414920 <structuredSignalHandler>,
handler_arg=0x1aa00, flags=506, result=0x7fedf5684cc8) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/omr/port/unix/omrsignal.c:425
#42 0x00007fedf552e6dc in gpProtectAndRun (function=0x7fedf54178e0 <gpProtectedRunCallInMethod(void*)>, env=0x1aa00, args=0x7fedf5684d30)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/util/jniprotect.c:78
#43 0x00007fedf54192b4 in gpCheckCallin (env=0x1aa00, receiver=receiver@entry=0x0, cls=0x119190, methodID=0x7fedf0616d38, args=args@entry=0x7fedf5684d78)
at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/jnicsup.cpp:489
#44 0x00007fedf54172ea in callStaticVoidMethod (env=<optimized out>, cls=<optimized out>, methodID=<optimized out>) at /root/CCM/JdkNext/openj9vt/openj9-openjdk-jdk.valuetypes/openj9/runtime/vm/jnicgen.c:384
#45 0x00007fedf58e18b6 in JavaMain (_args=<optimized out>) at src/java.base/share/native/libjli/java.c:551
#46 0x00007fedf58e4829 in ThreadJavaMain (args=<optimized out>) at src/java.base/unix/native/libjli/java_md.c:677
#47 0x00007fedf56a5609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#48 0x00007fedf5803293 in clone () from /lib/x86_64-linux-gnu/libc.so.6
It happens in the slow path of getFlattenedFieldAtOffset()
:
https://github.com/eclipse-openj9/openj9/blob/e1e0401e2fc38f0ef2b0307e91318907add666e9/runtime/vm/BytecodeInterpreter.hpp#L4027-L4036
The J9Method looks incorrect:
!J9StackWalkState 0x7fedf5683730
J9StackWalkState at 0x7fedf5683730 {
Fields for J9StackWalkState:
0x0: struct J9StackWalkState* previous = !j9stackwalkstate 0x00000000F56838E8
0x8: struct J9VMThread* walkThread = !j9vmthread 0x000000000001AA00
0x10: struct J9JavaVM* javaVM = !j9javavm 0x00007FEDF000F010
0x18: UDATA flags = 0x0000000024600002 (610271234)
0x20: UDATA* bp = !j9x 0x0000000000119078
0x28: UDATA* unwindSP = !j9x 0x0000000000000000
0x30: U8* pc = !j9x 0x00007FEDF55B14A4 // "�"
0x38: U8* nextPC = !j9x 0x00007FEDF55B14A4 // "�"
0x40: UDATA* sp = !j9x 0x0000000000119080
0x48: UDATA* arg0EA = !j9x 0x0000000000119080
0x50: struct J9Method* literals = !j9method 0x00007FEDD1C74C69 // <FAULT> < -------------- incorrect
0x58: UDATA* walkSP = !j9x 0x0000000000119080
0x60: UDATA argCount = 0x0000000000000000 (0)
0x68: struct J9ConstantPool* constantPool = !j9constantpool 0x0000009024B48B40 (flags = 0x0)
0x70: struct J9Method* method = !j9method 0x00007FEDD1C74C69 // <FAULT> < -------------- incorrect
0x78: struct J9JITExceptionTable* jitInfo = !j9jitexceptiontable 0x0000000000000000
...
And J9VMThread ->literals
is 0x8
!J9VMThread 0x1aa00
J9VMThread at 0x1aa00 {
Fields for J9VMThread:
0x0: struct JNINativeInterface_* functions = !jninativeinterface_ 0x00007FEDF55EBAC0
0x8: struct J9JavaVM* javaVM = !j9javavm 0x00007FEDF000F010
0x10: UDATA* arg0EA = !j9x 0x0000000000119078
0x18: UDATA* bytecodes = !j9x 0x0000000000000000
0x20: UDATA* sp = !j9x 0x0000000000119058
0x28: U8* pc = !j9x 0x0000000000000001
0x30: struct J9Method* literals = !j9method 0x0000000000000008 // <FAULT>
0x38: UDATA jitStackFrameFlags = 0x0000000040000000 (1073741824)
....
So the code built a special frame and update the vm thread, then calls getFlattenedFieldAtOffset()
in the slow path
and then push object in the the special frame before calling J9AllocateObject()
.
https://github.com/eclipse-openj9/openj9/blob/245cd10a22729c572ceb588b99f884cbddc49cad/runtime/vm/ValueTypeHelpers.hpp#L349-L352
Did you see anything incorrect here ? @tajila
Noticed currentThread->jitStackFrameFlags = 0x0000000040000000 (J9_SSF_JIT_NATIVE_TRANSITION_FRAME)
, not sure if this matters here.
The incorrect j9method 0x00007FEDD1C74C69
is from:
https://github.com/eclipse-openj9/openj9/blob/95981b1352fba3d7eb8787987acf4e9ad1424f34/runtime/vm/BytecodeInterpreter.hpp#L576
The method there is Unsafe.getValue()
, which we don't have JIT implementation yet: https://github.com/eclipse-openj9/openj9/issues/13696
@ehrenjulzert I think you should just test with -Xint
currently.
@hangshao0 @hzongaro
We should find out if the JIT will implement get/putValue
themselves or if they would want a fastJNI version. I imagine there are some cases (small fields) where the JIT would be able to do something very optimal, but it might be expensive to cover all cases, so there might be a need for a "fallback" implementation without transitioning to the interpreter.
We can revisit this later, as it currently depends on decisions on the JIT side.
We should find out if the JIT will implement get/putValue themselves or if they would want a fastJNI version.
Thanks for the reminder. We will take a look at this
Core dump: https://ibm.box.com/s/c5vptm6affrby6hcw1dym2jh9vks5mmz
This seems to be a JIT issue because when I run with
-Xint
the crash no longer happensStack trace
Reproducing
Branches used:
Code used
It should crash once
count
reaches about 1000000java arguments
java --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-opens java.base/jdk.internal.misc=ALL-UNNAMED -XX:ValueTypeFlatteningThreshold=999999 -XX:+EnableArrayFlattening -Xcompressedrefs
when the
-Xint
argument is added the crash no longer happens