search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JIT crash building vmState=0x000522ff #13450

Closed pshipton closed 6 months ago

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Build_JDK11_x86-64_windows_OpenJDK11/10

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK11_x86-64_windows_OpenJDK11/10/Build_JDK11_x86-64_windows_OpenJDK11-10-20210831-225113-diagnostics.tar.gz

23:49:50  Compiling 4 files for BUILD_JIGSAW_TOOLS
23:49:51  Generating javadoc for openj9 source files
23:49:53  Creating images/jmods/java.instrument.jmod
23:49:53  Creating images/jmods/java.datatransfer.jmod
23:49:53  Creating images/jmods/java.compiler.jmod
23:49:54  Creating images/jmods/java.desktop.jmod
23:50:00  Creating images/jmods/java.logging.jmod
23:50:07  Creating images/jmods/java.management.jmod
23:50:08  Creating images/jmods/java.management.rmi.jmod
23:50:08  Creating images/jmods/java.naming.jmod
23:50:12  Creating images/jmods/java.net.http.jmod
23:50:14  Creating images/jmods/java.prefs.jmod
23:50:18  Creating images/jmods/java.rmi.jmod
23:50:26  Creating images/jmods/java.scripting.jmod
23:50:28  Creating images/jmods/java.se.jmod
23:50:28  Creating images/jmods/java.security.jgss.jmod
23:50:32  Creating images/jmods/java.security.sasl.jmod
23:50:34  Creating images/jmods/java.smartcardio.jmod
23:50:37  Creating images/jmods/java.sql.jmod
23:50:45  Creating images/jmods/java.sql.rowset.jmod
23:50:46  Creating destination directory: "f:/Users/jenkins/workspace/Build_JDK11_x86-64_windows_OpenJDK11/build/windows-x86_64-normal-server-release/images/openj9-docs/api\"
23:50:48  Creating images/jmods/java.transaction.xa.jmod
23:50:48  Creating images/jmods/java.xml.jmod
23:50:53  Creating images/jmods/java.xml.crypto.jmod
23:50:55  Creating images/jmods/jdk.accessibility.jmod
23:50:56  Unhandled exception
23:50:56  Type=Segmentation error vmState=0x000522ff
23:50:56  Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFCF2F701A0 ContextFlags=0010005f
23:50:56  Handler1=00007FFCF2B7AE00 Handler2=00007FFCF401AAF0 InaccessibleReadAddress=0000000000000060
23:50:56  RDI=0000000000000E54 RSI=00007FF722365C60 RAX=00007FF71E430000 RBX=00007FF723A1DA90
23:50:56  RCX=0000000000000ED2 RDX=000000000000002B R8=00007FF7272BAAF8 R9=00007FF72257C320
23:50:56  R10=0000000000000003 R11=0000002AFAFA7F00 R12=00007FF722365C80 R13=00007FF7222E13A0
23:50:56  R14=00007FF727270000 R15=0000000000000000
23:50:56  RIP=00007FFCF2F701A0 RSP=0000002AFAFA7ED0 RBP=0000002AFAFA7F49 EFLAGS=0000000000010287
23:50:56  FS=0053 ES=002B DS=002B
23:50:56  XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM1 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
23:50:56  XMM2 ffffffffffffffff (f: 4294967296.000000, d: -nan)
23:50:56  XMM3 2929292929292929 (f: 690563392.000000, d: 2.092453e-110)
23:50:56  XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM7 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
23:50:56  XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:50:56  Module=f:\Users\jenkins\bootjdks\jdk11\bin\default\j9jit29.dll
23:50:56  Module_base_address=00007FFCF2B00000 Offset_in_DLL=00000000004701a0
23:50:56  
23:50:56  Method_being_compiled=com/sun/tools/javac/parser/JavadocTokenizer$JavadocComment.scanDocComment()V
23:50:56  Target=2_90_20210421_976 (Windows Server 2012 R2 6.3 build 9600)
23:50:56  CPU=amd64 (8 logical CPUs) (0x1ffb9c000 RAM)
23:50:56  ----------- Stack Backtrace -----------
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x39f400 (0x00007FFCF2F701A0 [j9jit29+0x4701a0])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x39e67a (0x00007FFCF2F6F41A [j9jit29+0x46f41a])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x39bd72 (0x00007FFCF2F6CB12 [j9jit29+0x46cb12])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x3a1973 (0x00007FFCF2F72713 [j9jit29+0x472713])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x2add72 (0x00007FFCF2E7EB12 [j9jit29+0x37eb12])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x36e319 (0x00007FFCF2F3F0B9 [j9jit29+0x43f0b9])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x36cdef (0x00007FFCF2F3DB8F [j9jit29+0x43db8f])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x36ac55 (0x00007FFCF2F3B9F5 [j9jit29+0x43b9f5])
23:50:56  Java_java_lang_invoke_MutableCallSite_invalidate+0x21de79 (0x00007FFCF2DEEC19 [j9jit29+0x2eec19])
23:50:56  (0x00007FFCF2B67A98 [j9jit29+0x67a98])
23:50:56  (0x00007FFCF2B6AD9B [j9jit29+0x6ad9b])
23:50:56  j9port_isCompatible+0x18a46 (0x00007FFCF401B576 [j9prt29+0x1b576])
23:50:56  j9port_isCompatible+0x1a184 (0x00007FFCF401CCB4 [j9prt29+0x1ccb4])
23:50:56  (0x00007FFCF2B6720D [j9jit29+0x6720d])
23:50:56  (0x00007FFCF2B6D22E [j9jit29+0x6d22e])
23:50:56  (0x00007FFCF2B6CC0A [j9jit29+0x6cc0a])
23:50:56  (0x00007FFCF2B7ABBF [j9jit29+0x7abbf])
23:50:56  j9port_isCompatible+0x1a1bf (0x00007FFCF401CCEF [j9prt29+0x1ccef])
23:50:56  (0x00007FFCF2B7A7DD [j9jit29+0x7a7dd])
23:50:56  omrthread_get_category+0xa42 (0x00007FFCF6204492 [j9thr29+0x4492])
23:50:56  _o_strcat_s+0x5e (0x00007FFCF437C1AE [ucrtbase+0x1c1ae])
23:50:56  BaseThreadInitThunk+0x22 (0x00007FFD03BC13F2 [KERNEL32+0x13f2])
23:50:56  RtlUserThreadStart+0x34 (0x00007FFD062154F4 [ntdll+0x154f4])
23:50:56  ---------------------------------------
23:50:56  JVMDUMP039I Processing dump event "gpf", detail "" at 2021/08/31 22:50:49 - please wait.
23:50:56  JVMDUMP032I JVM requested System dump using 'F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\core.20210831.225049.3544.0001.dmp' in response to an event
23:51:04  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\core.20210831.225049.3544.0001.dmp
23:51:04  JVMDUMP032I JVM requested Java dump using 'F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\javacore.20210831.225049.3544.0002.txt' in response to an event
23:51:04  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\javacore.20210831.225049.3544.0002.txt
23:51:04  JVMDUMP032I JVM requested Snap dump using 'F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\Snap.20210831.225049.3544.0003.trc' in response to an event
23:51:04  JVMDUMP010I Snap dump written to F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\Snap.20210831.225049.3544.0003.trc
23:51:04  JVMDUMP032I JVM requested JIT dump using 'F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\jitdump.20210831.225049.3544.0004.dmp' in response to an event
23:51:04  Creating images/jmods/jdk.attach.jmod
23:51:12  Creating images/jmods/jdk.charsets.jmod
23:51:12  JVMDUMP010I JIT dump written to F:\Users\jenkins\workspace\Build_JDK11_x86-64_windows_OpenJDK11\make\jitdump.20210831.225049.3544.0004.dmp
23:51:12  JVMDUMP013I Processed dump event "gpf", detail "".
23:51:12  make[3]: *** [/cygdrive/f/Users/jenkins/workspace/Build_JDK11_x86-64_windows_OpenJDK11/closed/custom/Docs.gmk:288: /cygdrive/f/Users/jenkins/workspace/Build_JDK11_x86-64_windows_OpenJDK11/build/windows-x86_64-normal-server-release/support/openj9-docs/_javadoc_openj9_only_exec.marker] Error 127
23:51:12  make[2]: *** [make/Main.gmk:427: docs-jdk-api-javadoc] Error 2
23:51:12  make[2]: *** Waiting for unfinished jobs....
23:51:17  
23:51:17  ERROR: Build failed for target 'all' in configuration 'windows-x86_64-normal-server-release' (exit code 2) 
23:51:17  Stopping sjavac server
pshipton commented 3 years ago

vmState [0x522ff]: {J9VMSTATE_JIT} {tacticalGlobalRegisterAllocator}

pshipton commented 3 years ago

A new build didn't repeat the crash. https://openj9-jenkins.osuosl.org/job/Build_JDK11_x86-64_windows_OpenJDK11/11

0xdaryl commented 3 years ago

@a7ehuo : could you investigate this GRA crash please?

0xdaryl commented 3 years ago

This may be related to MH's, however, due to the presence of Java_java_lang_invoke_MutableCallSite_invalidate in the call stack. But please triage why it is crashing in GRA.

a7ehuo commented 3 years ago

The crash happened while it was generating the build. There is no java executable in https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK11_x86-64_windows_OpenJDK11/10/Build_JDK11_x86-64_windows_OpenJDK11-10-20210831-225113-diagnostics.tar.gz

The jitdump shows the method was being compiled at hot. It has 170 inlined methods. The recompilation for jitdump is crashed in globalCopyPropagation due to std::bad_alloc. I suspect the recompilation passed the allocation limit and it is not related to the original crash in tacticalGlobalRegisterAllocator. 

<optimization id=68 name=globalCopyPropagation method=com/sun/tools/javac/parser/JavadocTokenizer$JavadocComment.scanDocComment()V>
...
...
[  8441] O^O COPY PROPAGATION:    Copy2 Propagation replacing Copy symRef #1610 
         O^O COPY PROPAGATION:    Use #2161[00007FF729572490] is defined by:
         O^O COPY PROPAGATION:       Def #260[00007FF71D115F00]
         O^O COPY PROPAGATION:    By Original expression in 00007FF729571270 Use node : 00007FF729572490

=== EXCEPTION THROWN (bad allocation) ===
</logRecompilation>
a7ehuo commented 3 years ago

Rebuild 4x jdk11 on Windows. All builds have passed. I'm still running more and see if it's reproducible.

2x with one change that removes TR_TraceAll and TR_EnableParanoidOptCheck in jitdmp (in order to see if we could get a complete log file if the crash in GRA happens again): 10483, 10497

2x with default master: 10492, 10502

a7ehuo commented 3 years ago

Another 9x builds passed. With the intermittent nature of this crash, it is hard to determine if it is related to MH or not

4x builds with one change that removes TR_TraceAll and TR_EnableParanoidOptCheck in jitdmp: 10513, 10515, 10524, 10527 5x builds with default master: 10514, 10516, 10519, 10525, 10526

a7ehuo commented 3 years ago

Another 12x builds passed: 6x builds with one change that removes TR_TraceAll and TR_EnableParanoidOptCheck in jitdmp; 6x builds with default master

a7ehuo commented 3 years ago

So far 25x builds have passed. Started another 10x builds and see if it's reproducible

a7ehuo commented 3 years ago

8x build passed. 2x builds were unfinished which are unrelated to this issue (10579, 10584)

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Build_JDK8_x86-32_windows_OMR/385

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Build_JDK8_x86-32_windows_OMR/385/Build_JDK8_x86-32_windows_OMR-385-20221207-053953-diagnostics.tar.gz

vmState [0x522ff]: {J9VMSTATE_JIT} {tacticalGlobalRegisterAllocator}

06:36:39  Structure stub generation complete
06:36:42  Unhandled structure type: ClassFileOracle$MethodIteratorPointer->_methodsInfo MethodInfo*
06:36:44  Unhandled structure type: GC_ObjectIteratorPointer->_type ScanType
06:36:44  Pointer class generation complete
06:37:11  Compiling 5400 files for j9ddr.jar
06:37:11  Compiling 816 files for BUILD_J9DDR_TEST_CLASSES
06:37:23  Unhandled exception
06:37:23  Type=Segmentation error vmState=0x000522ff
06:37:23  Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFA60BCF97F ContextFlags=0010005f
06:37:23  Handler1=00007FFA607BE230 Handler2=00007FFA61B4AB50 InaccessibleReadAddress=0000000000000060
06:37:23  RDI=00007FF7DE6B6A60 RSI=00007FF7E621DA88 RAX=00007FF7E12707B0 RBX=00007FF7E4FED3C0
06:37:23  RCX=0000000000000000 RDX=2000000000000000 R8=00007FF7DCF50000 R9=00007FF7E0B5A198
06:37:23  R10=3C00000000000006 R11=0000001EA16582B0 R12=00007FF7E24F0000 R13=00007FF7E61D0000
06:37:23  R14=0000000000000BD4 R15=0000000000000233
06:37:23  RIP=00007FFA60BCF97F RSP=0000001EA1658250 RBP=0000001EA1658350 EFLAGS=0000000000010293
06:37:23  FS=0053 ES=002B DS=002B
06:37:23  XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM1 4059000000000000 (f: 0.000000, d: 1.000000e+02)
06:37:23  XMM2 ffffffffffffffff (f: 4294967296.000000, d: -nan)
06:37:23  XMM3 2929292929292929 (f: 690563392.000000, d: 2.092453e-110)
06:37:23  XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM7 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
06:37:23  XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:37:23  Module=c:\cygwin64\home\jenkins\bootjdks\jdk8\jre\bin\default\j9jit29.dll
06:37:23  Module_base_address=00007FFA60750000 Offset_in_DLL=000000000047f97f
06:37:23  
06:37:23  Method_being_compiled=com/sun/tools/javac/parser/JavaTokenizer.readToken()Lcom/sun/tools/javac/parser/Tokens$Token;
06:37:23  Target=2_90_20221101_526 (Windows Server 2012 R2 6.3 build 9600)
06:37:23  CPU=amd64 (8 logical CPUs) (0x1ffb9c000 RAM)
06:37:23  ----------- Stack Backtrace -----------
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x3be75f (0x00007FFA60BCF97F [j9jit29+0x47f97f])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x3be352 (0x00007FFA60BCF572 [j9jit29+0x47f572])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x3c4034 (0x00007FFA60BD5254 [j9jit29+0x485254])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x2c77a2 (0x00007FFA60AD89C2 [j9jit29+0x3889c2])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x38f769 (0x00007FFA60BA0989 [j9jit29+0x450989])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x38e1f9 (0x00007FFA60B9F419 [j9jit29+0x44f419])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x38c035 (0x00007FFA60B9D255 [j9jit29+0x44d255])
06:37:23  Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk+0x232448 (0x00007FFA60A43668 [j9jit29+0x2f3668])
06:37:23  (0x00007FFA607AC214 [j9jit29+0x5c214])
06:37:23  (0x00007FFA607AF66F [j9jit29+0x5f66f])
06:37:23  j9port_isCompatible+0x18a96 (0x00007FFA61B4B5D6 [j9prt29+0x1b5d6])
06:37:23  j9port_isCompatible+0x1a1d4 (0x00007FFA61B4CD14 [j9prt29+0x1cd14])
06:37:23  (0x00007FFA607AB955 [j9jit29+0x5b955])
06:37:23  (0x00007FFA607B1C52 [j9jit29+0x61c52])
06:37:23  (0x00007FFA607B163A [j9jit29+0x6163a])
06:37:23  (0x00007FFA607BE064 [j9jit29+0x6e064])
06:37:23  j9port_isCompatible+0x1a20f (0x00007FFA61B4CD4F [j9prt29+0x1cd4f])
06:37:23  (0x00007FFA607BDD56 [j9jit29+0x6dd56])
06:37:23  omrthread_get_category+0xa42 (0x00007FFA63A44492 [j9thr29+0x4492])
06:37:23  _o_strcat_s+0x5e (0x00007FFA614CC1AE [ucrtbase+0x1c1ae])
06:37:23  BaseThreadInitThunk+0x22 (0x00007FFA714C13F2 [KERNEL32+0x13f2])
06:37:23  RtlUserThreadStart+0x34 (0x00007FFA73D854F4 [ntdll+0x154f4])
06:37:23  ---------------------------------------

@0xdaryl

0xdaryl commented 1 year ago

It is curious that this is a 64-bit JVM failure during a 32-bit build (likely a 64-bit JVM was produced during this process). Annabelle has had this on the backburner for over a year, but perhaps the new crash dump will provide fresh insights. Don't be fooled by the Java_java_lang_invoke_ThunkTuple_initialInvokeExactThunk in the backtrace--I suspect this is just a symbol issue.

@jdmpapin is working on scooping up the binaries for such build failures as we speak to make investigation easier.

pshipton commented 1 year ago

It is curious that this is a 64-bit JVM failure during a 32-bit build

Good point, it's the bootjdk (c:\cygwin64\home\jenkins\bootjdks\jdk8), which is the last release. This means we have the JVM and debug image. https://github.com/ibmruntimes/semeru8-binaries/releases/tag/jdk8u352-b08_openj9-0.35.0


openjdk version "1.8.0_352"
IBM Semeru Runtime Open Edition (build 1.8.0_352-b08)
Eclipse OpenJ9 VM (build openj9-0.35.0, JRE 1.8.0 Windows Server 2012 R2 amd64-64-Bit Compressed References 20221101_526 (JIT enabled, AOT enabled)
OpenJ9   - e04a7f6c1
OMR      - 85a21674f
JCL      - b213334935 based on jdk8u352-b08)
``
a7ehuo commented 1 year ago

The crash happened in TR_RegisterCandidate::processLiveOnEntryBlocks at line 772 [1] during the compilation of com/sun/tools/javac/parser/JavaTokenizer.readToken()Lcom/sun/tools/javac/parser/Tokens$Token; at scorching.

processLiveOnEntryBlocks loops through the predecessor edges of a block to get the from block. From what I can tell, it was processing predecessor edges of block_65. The edge is block_3028 -> block_65 [2]. Somehow either the from block is invalid or startOfExtendedBBForBB[...] is invalid. The jitdump collected after the crash completed successfully. The CFG edges from the jitdmp log don't seem to exactly match the ones during the crash: block_3028 doesn't reach block_65 [3].

[1] 

...
 767          TR::Block * block = blocks[blockNumber];
 768          bool referencedInPred = false;
 769          for (auto e = block->getPredecessors().begin(); e != block->getPredecessors().end(); ++e)
 770             {
 771             TR_ASSERT(comp->getOptimizer()->cachedExtendedBBInfoValid(), "Incorrect value in _startOfExtendedBBForBB");
 772             if (_liveOnEntry.get(startOfExtendedBBForBB[toBlock((*e)->getFrom())->getNumber()]->getNumber())) // <==== crash
0:005> kp
 # Child-SP          RetAddr           Call Site
00 (Inline Function) --------`-------- j9jit29!TR::CFGNode::getNumber(void)+0x4 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\infra\cfgnode.hpp @ 105] 
01 0000001e`a1658250 00007ffa`60bcf572 j9jit29!TR_RegisterCandidate::processLiveOnEntryBlocks(class TR::Block ** blocks = 0x00007ff7`e24f0000, int * blockStructureWeight = 0x00007ff7`e0a26d80, class TR::Compilation * comp = 0x00007ff7`e61d0000, class TR_Array<int> * blockGPRCount = 0x0000001e`a1658610, class TR_Array<int> * blockFPRCount = 0x0000001e`a1658638, class TR_Array<int> * blockVRFCount = 0x0000001e`a1658668, class TR_BitVector * referencedBlocks = 0x0000001e`a1658a50, class TR_Array<TR::Block *> * startOfExtendedBBForBB = 0x00007ff7`e621da88, bool callToRemoveUnusedLoops = false)+0x3df [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\registercandidate.cpp @ 772] 
02 0000001e`a1658420 00007ffa`60bd5254 j9jit29!TR_RegisterCandidate::setWeight(class TR::Block ** blocks = 0x00007ff7`e24f0000, int * blockStructureWeight = 0x00007ff7`e0a26d80, class TR::Compilation * comp = 0x00007ff7`e61d0000, class TR_Array<int> * blockGPRCount = 0x0000001e`a1658610, class TR_Array<int> * blockFPRCount = 0x0000001e`a1658638, class TR_Array<int> * blockVRFCount = 0x0000001e`a1658668, class TR_BitVector * referencedBlocks = 0x0000001e`a1658a50, class TR_Array<TR::Block *> * startOfExtendedBBForBB = 0x00007ff7`e621da88, class TR_BitVector * firstBlocks = 0x00007ff7`e621dba0, class TR_BitVector * isExtensionOfPreviousBlock = 0x00007ff7`e621dbc0)+0x732 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\registercandidate.cpp @ 632] 
03 0000001e`a1658530 00007ffa`60ad89c2 j9jit29!TR_RegisterCandidates::assign(class TR::Block ** cfgBlocks = 0x00007ff7`e24f0000, int numberOfBlocks = 0n3276, int * lowestNumber = 0x00007ff7`e688944c, int * highestNumber = 0x00007ff7`e6889450)+0x1c04 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\registercandidate.cpp @ 2305] 
04 0000001e`a1658de0 00007ffa`60ba0989 j9jit29!TR_GlobalRegisterAllocator::perform(void)+0x1382 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\globalregisterallocator.cpp @ 477] 
05 0000001e`a1658fe0 00007ffa`60b9f419 j9jit29!OMR::Optimizer::performOptimization(struct OptimizationStrategy * optimization = 0x00007ffa`60ebb0c8, int firstOptIndex = <Value unavailable error>, int lastOptIndex = <Value unavailable error>, int doTiming = 0n0)+0x18b9 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\omroptimizer.cpp @ 2053] 
06 0000001e`a165a280 00007ffa`60b9d255 j9jit29!OMR::Optimizer::performOptimization(struct OptimizationStrategy * optimization = 0x00007ffa`60e06fa0, int firstOptIndex = 0n0, int lastOptIndex = 0n2147483647, int doTiming = 0n0)+0x349 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\omroptimizer.cpp @ 1600] 
07 0000001e`a165b520 00007ffa`60a43668 j9jit29!OMR::Optimizer::optimize(void)+0x3e5 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\optimizer\omroptimizer.cpp @ 1129] 
08 (Inline Function) --------`-------- j9jit29!OMR::Compilation::performOptimizations(void)+0x26 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\compile\omrcompilation.cpp @ 1267] 
09 0000001e`a165ce50 00007ffa`607ac214 j9jit29!OMR::Compilation::compile(void)+0x698 [c:\workspace\openjdk-build\workspace\build\src\omr\compiler\compile\omrcompilation.cpp @ 1064] 
0a 0000001e`a165de90 00007ffa`607af66f j9jit29!TR::CompilationInfoPerThreadBase::compile(struct J9VMThread * vmThread = 0x00000000`00022300, class TR::Compilation * compiler = 0x00007ff7`e61d0000, class TR_ResolvedMethod * compilee = 0x0000001e`a165e6c8, class TR_J9VMBase * vm = 0x0000001e`a5e5aeb0, class TR_OptimizationPlan * optimizationPlan = 0x0000001e`a018b170, class TR::SegmentAllocator * scratchSegmentProvider = 0x0000001e`a165e570)+0x7e4 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 9648] 
0b 0000001e`a165e020 00007ffa`61b4b5d6 j9jit29!TR::CompilationInfoPerThreadBase::wrappedCompile(struct J9PortLibrary * portLib = 0x0000001e`9fc93d70, void * opaqueParameters = 0x0000001e`a165e4c0)+0x160f [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 9142] 
0c 0000001e`a165e1e0 00007ffa`61b4cd14 j9prt29!runInTryExcept(struct OMRPortLibrary * portLibrary = 0x00007ffa`63a9e700, <function> * fn = 0x00007ffa`00000000, void * fn_arg = 0x00007ff7`e61d0000, <function> * handler = 0x00007ffa`607be230, void * handler_arg = 0x00000000`00022300, unsigned int flags = 0x7d, unsigned int64 * result = 0x0000001e`a165e458)+0x16 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 220] 
0d 0000001e`a165e220 00007ffa`607ab955 j9prt29!omrsig_protect(struct OMRPortLibrary * portLibrary = 0x00007ffa`63a9e700, <function> * fn = 0x00007ffa`607ae060, void * fn_arg = 0x0000001e`a165e4c0, <function> * handler = 0x00007ffa`607be230, void * handler_arg = 0x00000000`00022300, unsigned int flags = 0x7d, unsigned int64 * result = 0x0000001e`a165e458)+0x214 [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 285] 
0e 0000001e`a165e400 00007ffa`607b1c52 j9jit29!TR::CompilationInfoPerThreadBase::compile(struct J9VMThread * vmThread = 0x00000000`00022300, struct TR_MethodToBeCompiled * entry = 0x0000001e`a5e7ba30, class J9::J9SegmentProvider * scratchSegmentProvider = 0x00000000`00022300)+0x3f5 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 8130] 
0f 0000001e`a165f6d0 00007ffa`607b163a j9jit29!TR::CompilationInfoPerThread::processEntry(struct TR_MethodToBeCompiled * entry = 0x0000001e`a5e7ba30, class J9::J9SegmentProvider * scratchSegmentProvider = 0x0000001e`a165f7c0)+0x342 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 4442] 
10 0000001e`a165f760 00007ffa`607be064 j9jit29!TR::CompilationInfoPerThread::processEntries(void)+0x14a [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 4136] 
11 (Inline Function) --------`-------- j9jit29!TR::CompilationInfoPerThread::run(void)+0x137 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 3980] 
12 0000001e`a165f850 00007ffa`61b4cd4f j9jit29!protectedCompilationThreadProc(struct J9PortLibrary * __formal = 0x0000001e`9fc93d70, class TR::CompilationInfoPerThread * compInfoPT = 0x0000001e`a00e2330)+0x284 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 3914] 
13 0000001e`a165f8a0 00007ffa`607bdd56 j9prt29!omrsig_protect(struct OMRPortLibrary * portLibrary = 0x00007ffa`63a9e700, <function> * fn = 0x00007ffa`607bdde0, void * fn_arg = 0x0000001e`a00e2330, <function> * handler = 0x00007ffa`6118cb70, void * handler_arg = 0x00000000`00022300, unsigned int flags = 0x7e, unsigned int64 * result = 0x0000001e`a165fae8)+0x24f [c:\workspace\openjdk-build\workspace\build\src\omr\port\win64amd\omrsignal.c @ 297] 
14 0000001e`a165fa80 00007ffa`63a44492 j9jit29!compilationThreadProc(void * entryarg = 0x0000001e`a00e2330)+0x276 [c:\workspace\openjdk-build\workspace\build\src\openj9\runtime\compiler\control\compilationthread.cpp @ 3822] 
15 0000001e`a165fae0 00007ffa`614cc1ae j9thr29!thread_wrapper(void * arg = 0x0000001e`9fc93d70)+0xf2 [c:\workspace\openjdk-build\workspace\build\src\omr\thread\common\omrthread.c @ 1737] 
16 0000001e`a165fb10 00007ffa`714c13f2 ucrtbase!thread_start<unsigned int +0x3e
17 0000001e`a165fb40 00007ffa`73d854f4 kernel32!BaseThreadInitThunk+0x22
18 0000001e`a165fb70 00000000`00000000 ntdll!RtlUserThreadStart+0x34

[2]

0000001e`a16582f8 class TR::Block * block = 0x00007ff7`e6307ca0

0:005> dx -r1 ((j9jit29!TR::CFGNode *)0x7ff7e6307ca0)
((j9jit29!TR::CFGNode *)0x7ff7e6307ca0)                 : 0x7ff7e6307ca0 [Type: OMR::Block * (derived from TR::CFGNode *)]
    [+0x008] _next            : 0x7ff7e63070e0 [Type: TR::CFGNode *]
    [+0x010] _valid           : true [Type: bool]
    [+0x018] _region          : 0x1ea165e620 [Type: TR::Region &]
    [+0x020] _successors      [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x030] _predecessors    [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x040] _exceptionSuccessors [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x050] _exceptionPredecessors [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x060] _nodeNumber      : 65 [Type: int]  // <==== block_65
    [+0x064] _visitCount      : 0x1be [Type: unsigned short]
    [+0x066] _frequency       : 347 [Type: short]
    [+0x068] _forwardTraversalIndex : 719 [Type: int]
    [+0x06c] _backwardTraversalIndex : 793 [Type: int]
    [=0x7ffa60fbe700] _standardExceptions [Type: OMR::Block::StandardException [0]]
    [+0x070] _pEntry          : 0x7ff7e6307db0 [Type: TR::TreeTop *]
    [+0x078] _pExit           : 0x7ff7e4fed4e0 [Type: TR::TreeTop *]
    [+0x080] _liveLocals      : 0x7ff7e3172860 [Type: TR_BitVector *]
    [+0x088] _pStructureOf    : 0x7ff7e091afe0 [Type: TR_BlockStructure *]
    [+0x090] _globalRegisters : 0x0 [Type: TR_Array<TR_GlobalRegister> *]
    [+0x098] _instructionBoundaries [Type: OMR::Block::InstructionBoundaries]
    [+0x0a8] _snippetBoundaries [Type: TR_LinkHead<OMR::Block::InstructionBoundaries>]
    [+0x0b0] _firstInstruction : 0x0 [Type: TR::Instruction *]
    [+0x0b8] _lastInstruction : 0x0 [Type: TR::Instruction *]
    [+0x0c0] _catchBlockExtension : 0x0 [Type: OMR::Block::TR_CatchBlockExtension *]
    [+0x0c8] _unrollFactor    : 0x0 [Type: unsigned short]
    [+0x0cc] _blockSize       : -1 [Type: int]
    [+0x0d0] _blockBCIndex    : 0 [Type: int]
    [+0x0d8] _j9EstimateSizeMethod : 0x0 [Type: TR_ResolvedMethod *]
    [+0x0e0] _debugCounters   : 0x0 [Type: TR::DebugCounterAggregation *]
    [+0x0e8] _flags           [Type: OMR::FlagsTemplate<unsigned int>]
    [+0x0ec] _moreflags       [Type: OMR::FlagsTemplate<unsigned int>]

dx -r1 ((j9jit29!TR::forward_list_private::ListElement<TR::CFGEdge *> *)0x7ff7e4fed3c0) // <==== CFGEdge *e

0:005> dx -r1 ((j9jit29!TR::forward_list_private::ListElement<TR::CFGEdge *> *)0x7ff7e4fed3c0)
((j9jit29!TR::forward_list_private::ListElement<TR::CFGEdge *> *)0x7ff7e4fed3c0)                 : 0x7ff7e4fed3c0 [Type: TR::forward_list_private::ListElement<TR::CFGEdge *> *]
    [+0x000] _next            : 0x7ff7e630d440 [Type: TR::forward_list_private::Link *]
    [+0x008] _value           : 0x7ff7e12707b0 [Type: TR::CFGEdge *]

0:005> dx -r1 ((j9jit29!TR::CFGEdge *)0x7ff7e12707b0)
((j9jit29!TR::CFGEdge *)0x7ff7e12707b0)                 : 0x7ff7e12707b0 [Type: TR::CFGEdge *]
    [+0x000] _next            : 0x0 [Type: TR::CFGEdge *]
    [+0x008] _pFrom           : 0x7ff7e4fed0d0 [Type: TR::CFGNode *] // <==== from block_3028
    [+0x010] _pTo             : 0x7ff7e6307ca0 [Type: TR::CFGNode *] //<==== to block_65
    [+0x018] _visitCount      : 0x0 [Type: unsigned short]
    [+0x01a] _flags           [Type: OMR::FlagsTemplate<unsigned short>]
    [+0x01c] _frequency       : 347 [Type: short]
    [+0x020] _id              : -1 [Type: int]

0:005> dx -r1 ((j9jit29!TR::CFGNode *)0x7ff7e4fed0d0)
((j9jit29!TR::CFGNode *)0x7ff7e4fed0d0)                 : 0x7ff7e4fed0d0 [Type: OMR::Block * (derived from TR::CFGNode *)]
    [+0x008] _next            : 0x7ff7e4fecdd0 [Type: TR::CFGNode *]
    [+0x010] _valid           : false [Type: bool]
    [+0x018] _region          : 0x1ea165e620 [Type: TR::Region &]
    [+0x020] _successors      [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x030] _predecessors    [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x040] _exceptionSuccessors [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x050] _exceptionPredecessors [Type: TR::forward_list<TR::CFGEdge *,TR::typed_allocator<TR::CFGEdge *,TR::Region &> >]
    [+0x060] _nodeNumber      : 3028 [Type: int] 
    [+0x064] _visitCount      : 0x0 [Type: unsigned short]
    [+0x066] _frequency       : 4379 [Type: short]
    [+0x068] _forwardTraversalIndex : -1 [Type: int]
    [+0x06c] _backwardTraversalIndex : -1 [Type: int]
    [=0x7ffa60fbe700] _standardExceptions [Type: OMR::Block::StandardException [0]]
    [+0x070] _pEntry          : 0x7ff7e4fed1c0 [Type: TR::TreeTop *]
    [+0x078] _pExit           : 0x7ff7e5102e50 [Type: TR::TreeTop *]
    [+0x080] _liveLocals      : 0x0 [Type: TR_BitVector *]
    [+0x088] _pStructureOf    : 0x0 [Type: TR_BlockStructure *]
    [+0x090] _globalRegisters : 0x0 [Type: TR_Array<TR_GlobalRegister> *]
    [+0x098] _instructionBoundaries [Type: OMR::Block::InstructionBoundaries]
    [+0x0a8] _snippetBoundaries [Type: TR_LinkHead<OMR::Block::InstructionBoundaries>]
    [+0x0b0] _firstInstruction : 0x0 [Type: TR::Instruction *]
    [+0x0b8] _lastInstruction : 0x0 [Type: TR::Instruction *]
    [+0x0c0] _catchBlockExtension : 0x0 [Type: OMR::Block::TR_CatchBlockExtension *]
    [+0x0c8] _unrollFactor    : 0x0 [Type: unsigned short]
    [+0x0cc] _blockSize       : -1 [Type: int]
    [+0x0d0] _blockBCIndex    : 0 [Type: int]
    [+0x0d8] _j9EstimateSizeMethod : 0x0 [Type: TR_ResolvedMethod *]
    [+0x0e0] _debugCounters   : 0x0 [Type: TR::DebugCounterAggregation *]
    [+0x0e8] _flags           [Type: OMR::FlagsTemplate<unsigned int>]
    [+0x0ec] _moreflags       [Type: OMR::FlagsTemplate<unsigned int>]

[3]

jitdump.20221207.053722.4532.0004.dmp

Trees before tactical global register allocator

        65 [0x00007FF7DB307CA0] BBStart at 0x00007FF7DB2EEFB0, frequency = 346
                 in        = [64(341) ]
                 out       = [3188(346) ]
                 exception in  = []
                 exception out = [121(6) ]
...
      3028 [0x00007FF7CA997700] BBStart at 0x00007FF7CB395230, frequency = 1
                 in        = [3029(1) ]
                 out       = [357(1) 3027(1) ]
                 exception in  = []
                 exception out = []
hzongaro commented 6 months ago

@a7ehuo, do you think we can close this issue as a duplicate of #18873?

pshipton commented 6 months ago

Since the last occurrence was Dec 2022, which was actually on the 35.0 release and not the current source, I don't have any issue closing, dup or not.

a7ehuo commented 6 months ago

Based on the backtrace, I'm pretty convinced this is the same issue as https://github.com/eclipse-openj9/openj9/issues/18873