Open JasonFengJ9 opened 1 year ago
OMRPortLibraryGlobalData at 0x7f89d0014e00 {
Fields for OMRPortLibraryGlobalData:
0x0: void* corruptedMemoryBlock = !j9x 0x0000600003E1E9B0
!whatis gives nothing useful for that address
1XMCURTHDINFO Current thread
3XMTHREADINFO "load-11" J9VMThread:0x000000000E9E5800, omrthread_t:0x00007F89D1893850, java/lang/Thread:0x00000000F04400A8, state:R, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x2A, isDaemon:false)
3XMJAVALTHRCCL jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000F0047660)
3XMTHREADINFO1 (native thread ID:0x58AA05F, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000021)
3XMTHREADINFO2 (native stack address range from:0x000070000D9EB000, to:0x000070000DA6B000, size:0x80000)
3XMCPUTIME CPU usage total: 4.556548000 secs, current category="Application"
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/invoke/MethodHandleNatives.resolve(Native Method)
4XESTACKTRACE at java/lang/invoke/MemberName$Factory.resolve(MemberName.java:1085(Compiled Code))
4XESTACKTRACE at java/lang/invoke/MemberName$Factory.resolveOrFail(MemberName.java:1114(Compiled Code))
4XESTACKTRACE at java/lang/invoke/MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3655(Compiled Code))
4XESTACKTRACE at java/lang/invoke/MethodHandles$Lookup.findVirtual(MethodHandles.java:2686(Compiled Code))
4XESTACKTRACE at net/adoptopenjdk/test/lang/invoke/AsTypeTest.testConvertLong(AsTypeTest.java:4794(Compiled Code))
4XESTACKTRACE at net/adoptopenjdk/test/lang/invoke/AsTypeTest.testLong(AsTypeTest.java:909)
lldb gives me a list of threads. I'm not sure how to find the one associated with the current thread above. The tid doesn't seem helpful. @knn-k do you know? By dumping them all I found #41 is the thread that found the memory corruption. Not all the frames are shown, I suspect the dSYM dirs aren't being found. Do you know how to fix that? All the files are in peter
on mac12x64rt13 (which is actually mac11x64rt13).
* thread #1: tid = 0x0000, 0x00007ff80dfa397a libsystem_kernel.dylib`mach_msg_trap + 10
thread #2: tid = 0x0001, 0x00007ff80dfa50ea libsystem_kernel.dylib`__ulock_wait + 10
thread #3: tid = 0x0002, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #4: tid = 0x0003, 0x00007ff80dfc7f3a libsystem_kernel.dylib`sem_wait + 10
thread #5: tid = 0x0004, 0x000000000665a2f3 libj9jit29.dylib`TR_ValueNumberInfo::allocateValueNumber(TR::Node*) [inlined] OMR::Node::getChild(this=0x00000000873bae70, c=<unavailable>) at OMRNode_inlines.hpp:207:14
thread #6: tid = 0x0005, 0x00000000064071d2 libj9jit29.dylib`OMR::Node::setRegister(this=0x0000000053a0f060, reg=0x00000000524d93b0) at OMRNode.cpp:4457:30
thread #7: tid = 0x0006, 0x000000000638bcea libj9jit29.dylib`OMR::CodeGenerator::cleanupFlags(TR::Node*) [inlined] OMR::Node::setVisitCount(this=0x0000000009c64690, vc=35) at OMRNode_inlines.hpp:282:24
thread #8: tid = 0x0007, 0x00000000065c640f libj9jit29.dylib`TR_RegisterCandidate::setWeight(TR::Block**, int*, TR::Compilation*, TR_Array<int>&, TR_Array<int>&, TR_Array<int>&, TR_BitVector*, TR_Array<TR::Block*>&, TR_BitVector&, TR_BitVector&) [inlined] TR_RegisterCandidate::symbolIsLive(this=0x000000007c85db01, block=0x000000000b32bb00) at RegisterCandidate.cpp:494:9
thread #9: tid = 0x0008, 0x00007ff80dff3629 libsystem_platform.dylib`_platform_bzero$VARIANT$Haswell + 41
thread #10: tid = 0x0009, 0x0000000006643eaf libj9jit29.dylib`TR_UseDefInfo::assignAdjustedNodeIndex(TR::Block*, TR::Node*, TR::Node*, TR::TreeTop*, TR_UseDefInfo::AuxiliaryData&, bool) [inlined] OMR::Node::setVisitCount(this=0x000000006f811590, vc=8504) at OMRNode_inlines.hpp:282:24
thread #11: tid = 0x000a, 0x00007ff80dff3659 libsystem_platform.dylib`_platform_bzero$VARIANT$Haswell + 89
thread #12: tid = 0x000b, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #13: tid = 0x000c, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #14: tid = 0x000d, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #15: tid = 0x000e, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #16: tid = 0x000f, 0x00007ff80dfaa092 libsystem_kernel.dylib`__accept + 10
thread #17: tid = 0x0010, 0x00007ff80dfaa092 libsystem_kernel.dylib`__accept + 10
thread #18: tid = 0x0011, 0x00007ff80dfaa092 libsystem_kernel.dylib`__accept + 10
thread #19: tid = 0x0012, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #20: tid = 0x0013, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #21: tid = 0x0014, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #22: tid = 0x0015, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #23: tid = 0x0016, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #24: tid = 0x0017, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #25: tid = 0x0018, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #26: tid = 0x0019, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #27: tid = 0x001a, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #28: tid = 0x001b, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #29: tid = 0x001c, 0x00007ff80dfc7f6a libsystem_kernel.dylib`semop + 10
thread #30: tid = 0x001d, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #31: tid = 0x001e, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #32: tid = 0x001f, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #33: tid = 0x0020, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #34: tid = 0x0021, 0x0000000023a7d1cf
thread #35: tid = 0x0022, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #36: tid = 0x0023, 0x000000000438a488 libj9vm29.dylib`VM_BytecodeInterpreterCompressed::run(this=0x000070000d7d9ae0, vmThread=<unavailable>) at BytecodeInterpreter.hpp:0
thread #37: tid = 0x0024, 0x0000000023c9065d
thread #38: tid = 0x0025, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #39: tid = 0x0026, 0x000000002387b132
thread #40: tid = 0x0027, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #41: tid = 0x0028, 0x00007ff80dfa9152 libsystem_kernel.dylib`__wait4 + 10
thread #42: tid = 0x0029, 0x0000000004322e75 libj9vm29.dylib`::j9jni_createLocalRef(env=0x000000000e9e9400, object=0x00000000ff77cc98) at jnicsup.cpp:908:29
thread #43: tid = 0x002a, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #44: tid = 0x002b, 0x00007ff80debc8c2 libsystem_c.dylib`__Balloc_D2A + 243
thread #45: tid = 0x002c, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #46: tid = 0x002d, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #47: tid = 0x002e, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #48: tid = 0x002f, 0x000000002388ca23
thread #49: tid = 0x0030, 0x0000000023a8ea57
thread #50: tid = 0x0031, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #51: tid = 0x0032, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #52: tid = 0x0033, 0x000000000438bd16 libj9vm29.dylib`VM_BytecodeInterpreterCompressed::run(this=0x000000000e30dac0, vmThread=<unavailable>) at BytecodeInterpreter.hpp:0
thread #53: tid = 0x0034, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #54: tid = 0x0035, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #55: tid = 0x0036, 0x0000000023a8eaf7
thread #56: tid = 0x0037, 0x0000000007302000 libj9jit29.dylib`jitExitInterpreterJ
thread #57: tid = 0x0038, 0x0000000023656438
thread #58: tid = 0x0039, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #59: tid = 0x003a, 0x00007ff80dfa3a82 libsystem_kernel.dylib`swtch_pri + 10
thread #60: tid = 0x003b, 0x00007ff80dfaa0aa libsystem_kernel.dylib`poll + 10
thread #61: tid = 0x003c, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #62: tid = 0x003d, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
thread #63: tid = 0x003e, 0x00007ff80dfaa0aa libsystem_kernel.dylib`poll + 10
thread #64: tid = 0x003f, 0x00007ff80dfa63ea libsystem_kernel.dylib`__psynch_cvwait + 10
!j9thread 0x00007F89D1893850
J9Thread at 0x7f89d1893850 {
Fields for J9Thread:
0x0: struct J9ThreadLibrary* library = !j9threadlibrary 0x0000000003FD7838
0x8: U64 attachcount = 0x0000000000000000 (0)
0x10: U64 priority = 0x0000000000000005 (5)
0x18: struct J9ThreadMonitor* monitor = !j9threadmonitor 0x0000000000000000
0x20: struct J9Thread* next = !j9thread 0x0000000000000000
0x28: struct J9Thread* prev = !j9thread 0x0000000000000000
0x30: void*[] tls = !j9x 0x00007F89D1893880
0x410: void* entrypoint = !j9x 0x000000000436FC80
0x418: void* entryarg = !j9x 0x00007F89D001B220
0x420: U64 flags = 0x0000000001000800 (16779264)
0x428: U64 tid = 0x00000000058AA05F (92971103)
0x430: struct J9Thread* interrupter = !j9thread 0x0000000000000000
0x438: U32 category = 0x00000100 (256)
0x43c: U32 effective_category = 0x00000100 (256)
0x440: I64 lastCategorySwitchTime = 0x0000000000000000 (0)
0x448: struct J9ThreadTracing* tracing = !j9threadtracing 0x00007F89D00CA448
0x450: U64 waitNumber = 0x0000000000000000 (0)
0x458: U64 lockedmonitorcount = 0x0000000000000000 (0)
0x460: I64 os_errno = 0xFFFFFFFFFFFFFFFF (-1)
0x468: struct _opaque_pthread_t* handle = !_opaque_pthread_t 0x000070000DA6B000
0x470: struct _opaque_pthread_cond_t condition = !_opaque_pthread_cond_t 0x00007F89D1893CC0
0x4a0: struct _opaque_pthread_mutex_t mutex = !_opaque_pthread_mutex_t 0x00007F89D1893CF0
0x4e0: U64 stacksize = 0x0000000000080000 (524288)
0x4e8: U8[] numaAffinity = !j9x 0x00007F89D1893D38
0x568: struct J9ThreadMonitor* destroyed_monitor_head = !j9threadmonitor 0x0000000000000000
0x570: struct J9ThreadMonitor* destroyed_monitor_tail = !j9threadmonitor 0x0000000000000000
0x578: U64 key_deletion_attempts = 0x0000000000000000 (0)
}
thread #41
frame #0: 0x00007ff80dfa9152 libsystem_kernel.dylib`__wait4 + 10
frame #1: 0x000000000406cffe libj9prt29.dylib`omrdump_create(portLibrary=<unavailable>, filename="/Users/jenkins/workspace/Test_openjdk17_j9_sanity.system_x86-64_mac_testList_0/aqa-tests/TKG/output_16733873544924/TestJlmRemoteMemoryAuth_0/20230110-172859-TestJlmRemoteMemoryAuth/results/core.20230110.173445.17632.0001.dmp", dumpType=<unavailable>, userData=<unavailable>) at omrosdump.c:679:3 [opt]
frame #2: 0x000000000414b9e8 libj9dmp29.dylib`doSystemDump(agent=<unavailable>, label="/Users/jenkins/workspace/Test_openjdk17_j9_sanity.system_x86-64_mac_testList_0/aqa-tests/TKG/output_16733873544924/TestJlmRemoteMemoryAuth_0/20230110-172859-TestJlmRemoteMemoryAuth/results/core.20230110.173445.17632.0001.dmp", context=0x000070000da682f8) at dmpagent.c:756:12 [opt]
frame #3: 0x0000000004150ae5 libj9dmp29.dylib`protectedDumpFunction(portLibrary=<unavailable>, userData=<unavailable>) at dmpagent.c:2852:16 [opt]
frame #4: 0x000000000406e672 libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000003f9d1d8, fn=<unavailable>, fn_arg=0x000070000da67e68, handler=<unavailable>, handler_arg=0x0000000000000000, flags=<unavailable>, result=0x000070000da67e58) at omrsignal.c:425:12 [opt]
frame #5: 0x000000000415071b libj9dmp29.dylib`runDumpAgent [inlined] runDumpFunction(agent=0x0000600002834260, label="/Users/jenkins/workspace/Test_openjdk17_j9_sanity.system_x86-64_mac_testList_0/aqa-tests/TKG/output_16733873544924/TestJlmRemoteMemoryAuth_0/20230110-172859-TestJlmRemoteMemoryAuth/results/core.20230110.173445.17632.0001.dmp", context=0x000070000da682f8) at dmpagent.c:2830:21 [opt]
frame #6: 0x00000000041506cc libj9dmp29.dylib`runDumpAgent(vm=0x00007f89d001b220, agent=<unavailable>, context=0x000070000da682f8, state=<unavailable>, detail="", timeNow=<unavailable>) at dmpagent.c:2760:13 [opt]
frame #7: 0x0000000004169194 libj9dmp29.dylib`triggerDumpAgents(vm=<unavailable>, self=<unavailable>, eventFlags=<unavailable>, eventData=<unavailable>) at trigger.c:1036:7 [opt]
frame #8: 0x00000000041c9504
frame #9: 0x00000000041bd99f
frame #10: 0x00000000041b9905
frame #11: 0x0000000004067994 libj9prt29.dylib`unwrapBlockAndCheckTags(portLibrary=<unavailable>, memoryPointer=<unavailable>) at omrmemtag.c:145:3 [opt]
frame #12: 0x0000000004067825 libj9prt29.dylib`omrmem_free_memory(portLibrary=0x0000000003f9d1d8, memoryPointer=<unavailable>) at omrmemtag.c:208:19 [opt]
frame #13: 0x000000000495b659
frame #14: 0x0000000023adbc29
frame #15: 0x00000000042fa954 libj9vm29.dylib`::runJavaThread(currentThread=<unavailable>) at callin.cpp:682:4 [opt]
frame #16: 0x000000000436fdb4 libj9vm29.dylib`javaProtectedThreadProc(portLibrary=<unavailable>, entryarg=0x000000000e9e5800) at vmthread.cpp:2093:3 [opt]
frame #17: 0x000000000406e672 libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000003f9d1d8, fn=<unavailable>, fn_arg=0x000000000e9e5800, handler=<unavailable>, handler_arg=0x000000000e9e5800, flags=<unavailable>, result=0x000070000da68f70) at omrsignal.c:425:12 [opt]
frame #18: 0x000000000436fcc8 libj9vm29.dylib`::javaThreadProc(entryarg=0x00007f89d001b220) at vmthread.cpp:372:2 [opt]
frame #19: 0x0000000003fd163a
frame #20: 0x00007ff80dfe04e1 libsystem_pthread.dylib`_pthread_start + 125
frame #21: 0x00007ff80dfdbf6b libsystem_pthread.dylib`thread_start + 15
corruptedMemoryBlock = !j9x 0x0000600003E1E9B0
!j9x 0x0000600003E1E990,120
0x600003E1E990 : 0000bce906b7e990 000000000000008a [ ................ ]
0x600003E1E9A0 : 0000000004566623 0000000003f9bdb0 [ #fV............. ]
0x600003E1E9B0 : 0000747365740004 ddddc703badbad21 [ ..test..!....... ]
0x600003E1E9C0 : be95ff7cbadbad21 000000000000000e [ !...|........... ]
0x600003E1E9D0 : 0000000004566623 0000000003f9bdb0 [ #fV............. ]
0x600003E1E9E0 : 66ca8c3fb1234567 0000000000000006 [ gE#.?..f........ ]
0x600003E1E9F0 : 0000000004096777 00007f89d0015840 [ wg......@X...... ]
0x600003E1EA00 : dddd00746e657665 608c8991b7654321 [ event...!Ce....` ]
0x600003E1EA10 : 0000000000000006 0000000004096777 [ ........wg...... ]
0x600003E1EA20 : 00007f89d0015840 0000000003f9bdb0 [ @X.............. ]
0x600003E1EA30 : 66ca8fecb1234567 0000000000000005 [ gE#....f........ ]
0x600003E1EA40 : 0000000004096777 00007f89d0015840 [ wg......@X...... ]
0x600003E1EA50 : dddddd007473696c 608c89c2b7654321 [ list....!Ce....` ]
0x600003E1EA60 : 0000000000000005 0000000004096777 [ ........wg...... ]
0x600003E1EA70 : 00007f89d0015840 0000000000000000 [ @X.............. ]
0x600003E1EA80 : 66ca8f5fb1234567 0000000000000006 [ gE#._..f........ ]
0x600003E1EA90 : 0000000004096777 00007f89d0015840 [ wg......@X...... ]
0x600003E1EAA0 : dddd006e69626d76 608c8931b7654321 [ vmbin...!Ce.1..` ]
@knn-k are you able to take a look? See my comments in https://github.com/eclipse-openj9/openj9/issues/16544#issuecomment-1380919199
The tid doesn't seem helpful. @knn-k do you know?
I don't know what lldb prints as tid. There is no field in the related macOS structs for storing the thread ID as far as I know.
Not all the frames are shown, I suspect the dSYM dirs aren't being found. Do you know how to fix that?
I'm afraid I don't know it, either.
I looked into the core file of the failure, but I haven't got any clue.
Failure link
From an internal build(
mac12x64rt13
):Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
aqa-tests/TKG/output_16733873544924/TestJlmRemoteMemoryAuth_0/20230110-172859-TestJlmRemoteMemoryAuth/results/7.LT2.stderr
30x internal grinder - all passed