search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 720 forks source link

OpenJDK java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned Invalid JIT return address ASSERTION FAILED swalk.c:1629: ((0 )) #18910

Closed pshipton closed 3 weeks ago

pshipton commented 9 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/1 jdk_lang_1 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/1/openjdk_test_output.tar.gz

21:44:10  *** Invalid JIT return address 48000000D024848B in 00007000087988C0
21:44:10  
21:44:10  02:42:41.635 0x7f81f2854900    j9vm.249    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK22_x86-64_mac_Nightly/openj9/runtime/vm/swalk.c:1629: ((0 ))
pshipton commented 9 months ago

Perhaps related to https://github.com/eclipse-openj9/openj9/issues/18908

hzongaro commented 8 months ago

Saw a failure with a different symptom in an internal grinder run https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/37939:

01:02:45  2024-02-13T06:02:19.930075Z 359252 remaining ...
01:02:45  2024-02-13T06:02:20.431950Z 356836 remaining ...
01:02:45  STDERR:
01:02:45  Unhandled exception
01:02:45  Type=Segmentation error vmState=0x00000000
01:02:45  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
01:02:45  Handler1=00000000068380E0 Handler2=000000000272F7C0 InaccessibleAddress=0000000000000009
01:02:45  RDI=00007F93CD6140E8 RSI=00000000B3A9E6E9 RAX=0000000000000008 RBX=00007000097388C0
01:02:45  RCX=00000000148C0100 RDX=0000000000000001 R8=0000000000000FDE R9=0000000000001009
01:02:45  R10=00000000FFFFFE00 R11=0000000000000000 R12=000000000F8161E8 R13=00000000140C0100
01:02:45  R14=0000000000000006 R15=0000700009738920
01:02:45  RIP=000000000687A6B9 GS=0000 FS=0000 RSP=0000700009738820
01:02:45  RFlags=0000000000010203 CS=002B RBP=00007000097388A0 ERR=0000000900000004
01:02:45  TRAPNO=000000040000000E CPU=0009000000040000 FAULTVADDR=0000000000000009
01:02:45  XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM1 000000000f7cb6d8 (f: 259831520.000000, d: 1.283738e-315)
01:02:45  XMM2 000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
01:02:45  XMM3 3e02b34ff0f2b4e0 (f: 4042437888.000000, d: 5.442544e-10)
01:02:45  XMM4 3ea7435e9e8af299 (f: 2659906304.000000, d: 6.932963e-07)
01:02:45  XMM5 00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
01:02:45  XMM6 3fd31871c9544185 (f: 3377742336.000000, d: 2.983670e-01)
01:02:45  XMM7 402791272ee9dd8e (f: 787078528.000000, d: 1.178350e+01)
01:02:45  XMM8 0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
01:02:45  XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:02:45  Module=/Users/jenkins/workspace/Grinder_iteration_2/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
01:02:45  Module_base_address=0000000006800000 Symbol=walkStackFrames
01:02:45  Symbol_address=0000000006879F70
01:02:45  Target=2_90_20240205_7 (Mac OS X 11.2.3)
01:02:45  CPU=amd64 (8 logical CPUs) (0x400000000 RAM)
01:02:45  ----------- Stack Backtrace -----------
01:02:45  walkStackFrames+0x74a (0x000000000687A6BA [libj9vm29.dylib+0x7a6ba])
01:02:45  getStackTraceForThread+0x20d (0x0000000008E3E41D [libjclse29.dylib+0xa41d])
01:02:45  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x0000000008E77D11 [libjclse29.dylib+0x43d11])
01:02:45  ---------------------------------------
01:02:45  JVMDUMP039I Processing dump event "gpf", detail "" at 2024/02/13 01:02:20 - please wait.
01:02:45  JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/core.20240213.010220.62695.0001.dmp' in response to an event
01:02:45  JVMDUMP010I System dump written to /Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/core.20240213.010220.62695.0001.dmp
01:02:45  JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/javacore.20240213.010220.62695.0002.txt' in response to an event
01:02:45  JVMDUMP010I Java dump written to /Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/javacore.20240213.010220.62695.0002.txt
01:02:45  JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/Snap.20240213.010220.62695.0003.trc' in response to an event
01:02:45  JVMDUMP010I Snap dump written to /Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/Snap.20240213.010220.62695.0003.trc
01:02:45  JVMDUMP032I JVM requested JIT dump using '/Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/jitdump.20240213.010220.62695.0004.dmp' in response to an event
01:02:45  JVMDUMP051I JIT dump occurred in 'MainThread' thread 0x000000000F772F00
01:02:45  JVMDUMP053I JIT dump is recompiling java/lang/VirtualThread.asyncGetStackTrace()[Ljava/lang/StackTraceElement;
01:02:45  JVMDUMP010I JIT dump written to /Users/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_17078040982155/jdk_custom_0_ITER_1/work/scratch/0/jitdump.20240213.010220.62695.0004.dmp
01:02:45  JVMDUMP013I Processed dump event "gpf", detail "".
pshipton commented 8 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/15/ jdk_lang_1 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/15/openjdk_test_output.tar.gz

21:22:12  Type=Segmentation error vmState=0x00000000
21:22:12  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
21:22:12  Handler1=000000000DB22070 Handler2=000000000D878790 InaccessibleAddress=0000000000000009
21:22:12  RDI=00007FD9C970AA98 RSI=00000000B7C9D080 RAX=0000000000000008 RBX=000070000B2998C0
21:22:12  RCX=00000000148C0100 RDX=0000000000000001 R8=0000000000000066 R9=00000000FFFFFFF0
21:22:12  R10=0000000000FFFFFF R11=0000000000000000 R12=00007FD9C988D7F8 R13=00000000140C0100
21:22:12  R14=0000000000000006 R15=000070000B299920
21:22:12  RIP=000000000DB64669 GS=0000 FS=0000 RSP=000070000B299820
21:22:12  RFlags=0000000000010207 CS=002B RBP=000070000B2998A0 ERR=0000000900000004
21:22:12  TRAPNO=000000040000000E CPU=0009000000040000 FAULTVADDR=0000000000000009
21:22:12  XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM1 0080000100800001 (f: 8388609.000000, d: 2.848097e-306)
21:22:12  XMM2 000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
21:22:12  XMM3 3e3ca4f39d0c5e80 (f: 2634833408.000000, d: 6.669281e-09)
21:22:12  XMM4 3ecee9789b43759a (f: 2604889600.000000, d: 3.684997e-06)
21:22:12  XMM5 0000000000200000 (f: 2097152.000000, d: 1.036131e-317)
21:22:12  XMM6 3f87dc475f810a77 (f: 1602292352.000000, d: 1.165062e-02)
21:22:12  XMM7 402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
21:22:12  XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:22:12  Module=/Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
21:22:12  Module_base_address=000000000DAEA000 Symbol=walkStackFrames
21:22:12  Symbol_address=000000000DB63F20
21:22:12  Target=2_90_20240227_26 (Mac OS X 14.3)
21:22:12  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
21:22:12  ----------- Stack Backtrace -----------
21:22:12  walkStackFrames+0x74a (0x000000000DB6466A [libj9vm29.dylib+0x7a66a])
21:22:12  getStackTraceForThread+0x20d (0x000000000E11D45D [libjclse29.dylib+0xa45d])
21:22:12  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000E156D51 [libjclse29.dylib+0x43d51])
21:22:12  ---------------------------------------
pshipton commented 7 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/28 jdk_lang_0 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/28/openjdk_test_output.tar.gz

21:30:04  Type=Segmentation error vmState=0x00000000
21:30:04  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
21:30:04  Handler1=0000000001F16280 Handler2=0000000001C6C790
21:30:04  RDI=00007FCAE484B050 RSI=000000001B337CD5 RAX=41000000A024A480 RBX=0000700010F6A8C0
21:30:04  RCX=0000000000000000 RDX=4C44618B45C38B4C R8=0000000000001AD8 R9=AA9ECF6CDAE52767
21:30:04  R10=0000000000003F01 R11=00007FCAE4800000 R12=0000000005926170 R13=00000000140C0100
21:30:04  R14=B3BB9E74BC531227 R15=0000000001B95FE8
21:30:04  RIP=0000000001F58AA1 GS=0000 FS=0000 RSP=0000700010F6A820
21:30:04  RFlags=0000000000010297 CS=002B RBP=0000700010F6A8A0 ERR=2345600000000000
21:30:04  TRAPNO=000000000000000D CPU=6000000000000000 FAULTVADDR=0000000123456000
21:30:04  XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM1 00000000058ce748 (f: 93120328.000000, d: 4.600755e-316)
21:30:04  XMM2 000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
21:30:04  XMM3 3e43c894bc460f06 (f: 3158707968.000000, d: 9.212419e-09)
21:30:04  XMM4 3ed32b93127d7f18 (f: 310214432.000000, d: 4.570535e-06)
21:30:04  XMM5 00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
21:30:04  XMM6 3fcdc1bca0abec7d (f: 2695621632.000000, d: 2.324749e-01)
21:30:04  XMM7 402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
21:30:04  XMM8 0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
21:30:04  XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:30:04  Module=/Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
21:30:04  Module_base_address=0000000001EDE000 Symbol=walkStackFrames
21:30:04  Symbol_address=0000000001F58190
21:30:04  Target=2_90_20240318_40 (Mac OS X 13.4.1)
21:30:04  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
21:30:04  ----------- Stack Backtrace -----------
21:30:04  walkStackFrames+0x912 (0x0000000001F58AA2 [libj9vm29.dylib+0x7aaa2])
21:30:04  getStackTraceForThread+0x20d (0x00000000025114ED [libjclse29.dylib+0xa4ed])
21:30:04  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000254ADF1 [libjclse29.dylib+0x43df1])
vij-singh commented 6 months ago

Any new news on this one?

hzongaro commented 6 months ago

Any new news on this one?

I haven't been able to make further progress on this one.

pshipton commented 6 months ago

@hzongaro we won't hold the release for this, pls move it out of the Java 22 milestone.

JasonFengJ9 commented 6 months ago

JDK22 x86-64_mac(macx64rt5)

openjdk version "22.0.1" 2024-04-16
IBM Semeru Runtime Open Edition 22.0.1.0-m2 (build 22.0.1+8)
Eclipse OpenJ9 VM 22.0.1.0-m2 (build v0.45.0-release-349ff6eab, JRE 22 Mac OS X amd64-64-Bit Compressed References 20240416_49 (JIT enabled, AOT enabled)
OpenJ9   - 349ff6eab
OMR      - 254af5a04
JCL      - b3c38dcd1fd based on jdk-22.0.1+8)

[2024-04-23T18:09:30.670Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
[2024-04-23T18:09:30.670Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops -Xverbosegclog 

[2024-04-23T18:15:35.855Z] TEST: java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

[2024-04-23T18:15:35.856Z] STDERR:
[2024-04-23T18:15:35.856Z] Unhandled exception
[2024-04-23T18:15:35.856Z] Type=Segmentation error vmState=0x00000000
[2024-04-23T18:15:35.856Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
[2024-04-23T18:15:35.856Z] Handler1=00000000104DD4C0 Handler2=000000001022C9E0
[2024-04-23T18:15:35.856Z] RDI=00007FC0D59BB458 RSI=000000001296EEEA RAX=B3894C0000010080 RBX=000070000B4108C0
[2024-04-23T18:15:35.856Z] RCX=00000000148C0100 RDX=00007FC0D789C7C8 R8=000070000B4107A0 R9=0000000010155FE8
[2024-04-23T18:15:35.856Z] R10=0000000000000480 R11=00007FC0C390F9C5 R12=00007FC0D789C7C8 R13=000070000B4108C0
[2024-04-23T18:15:35.856Z] R14=89481A7703F88348 R15=000070000B410920
[2024-04-23T18:15:35.856Z] RIP=000000001051FE9C GS=0000 FS=0000 RSP=000070000B410820
[2024-04-23T18:15:35.856Z] RFlags=0000000000010282 CS=002B RBP=000070000B4108A0 ERR=1023F00000000000
[2024-04-23T18:15:35.856Z] TRAPNO=000000000000000D CPU=F000000000000000 FAULTVADDR=000000001023F000
[2024-04-23T18:15:35.856Z] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM1 00007fc0d4046d18 (f: 3557059840.000000, d: 6.939951e-310)
[2024-04-23T18:15:35.856Z] XMM2 000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
[2024-04-23T18:15:35.856Z] XMM3 3e1d2d3b29444e10 (f: 692342272.000000, d: 1.698307e-09)
[2024-04-23T18:15:35.856Z] XMM4 3eb8d680c731cbde (f: 3341929472.000000, d: 1.480454e-06)
[2024-04-23T18:15:35.856Z] XMM5 0000000000200000 (f: 2097152.000000, d: 1.036131e-317)
[2024-04-23T18:15:35.856Z] XMM6 3fdbc1e08b0dad0a (f: 2332929280.000000, d: 4.337083e-01)
[2024-04-23T18:15:35.856Z] XMM7 402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
[2024-04-23T18:15:35.856Z] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-04-23T18:15:35.856Z] Module=/Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-04-23T18:15:35.856Z] Module_base_address=00000000104A5000 Symbol=walkStackFrames
[2024-04-23T18:15:35.856Z] Symbol_address=000000001051F320
[2024-04-23T18:15:35.856Z] Target=2_90_20240416_49 (Mac OS X 13.2.1)
[2024-04-23T18:15:35.856Z] CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
[2024-04-23T18:15:35.856Z] ----------- Stack Backtrace -----------
[2024-04-23T18:15:35.856Z] walkStackFrames+0xb7d (0x000000001051FE9D [libj9vm29.dylib+0x7ae9d])
[2024-04-23T18:15:35.856Z] getStackTraceForThread+0x20d (0x0000000010B1F94D [libjclse29.dylib+0x894d])
[2024-04-23T18:15:35.856Z] Java_java_lang_Thread_getStackTraceImpl+0x51 (0x0000000010B59251 [libjclse29.dylib+0x42251])
[2024-04-23T18:15:35.856Z] ---------------------------------------
[2024-04-23T18:15:35.856Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/04/23 14:13:59 - please wait.

[2024-04-23T18:15:35.856Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
[2024-04-23T18:15:35.856Z] --------------------------------------------------
[2024-04-23T18:16:46.796Z] Test results: passed: 927; failed: 1
[2024-04-23T18:16:54.595Z] Report written to /Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/aqa-tests/TKG/output_1713895769342/jdk_lang_1/report/html/report.html
[2024-04-23T18:16:54.595Z] Results written to /Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/aqa-tests/TKG/output_1713895769342/jdk_lang_1/work
[2024-04-23T18:16:54.595Z] Error: Some tests failed or other problems occurred.
[2024-04-23T18:16:54.595Z] -----------------------------------
[2024-04-23T18:16:54.595Z] jdk_lang_1_FAILED

50x grinder - passed

JasonFengJ9 commented 5 months ago

JDK22 x86-64_mac(macx64rt6)

[2024-05-14T02:37:01.982Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
[2024-05-14T02:37:01.982Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-

[2024-05-14T02:43:33.923Z] TEST: java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

[2024-05-14T02:43:33.924Z] STDERR:
[2024-05-14T02:43:33.924Z] Unhandled exception
[2024-05-14T02:43:33.924Z] Type=Segmentation error vmState=0x00000000
[2024-05-14T02:43:33.924Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
[2024-05-14T02:43:33.924Z] Handler1=000000000216C880 Handler2=0000000001EBB9D0
[2024-05-14T02:43:33.924Z] RDI=00007FE834A54488 RSI=00000000046056CA RAX=B3894C0000010080 RBX=0000700006B5B8C0
[2024-05-14T02:43:33.924Z] RCX=00000000148C0100 RDX=00007FE83409F3B8 R8=000000000000F98B R9=331BC3C45606744E
[2024-05-14T02:43:33.924Z] R10=0000000000003F01 R11=00007FE834800000 R12=00007FE83409F3B8 R13=0000700006B5B8C0
[2024-05-14T02:43:33.924Z] R14=89481A7703F88348 R15=0000700006B5B920
[2024-05-14T02:43:33.924Z] RIP=00000000021AF19C GS=0000 FS=0000 RSP=0000700006B5B820
[2024-05-14T02:43:33.924Z] RFlags=0000000000010282 CS=002B RBP=0000700006B5B8A0 ERR=2D92500000000000
[2024-05-14T02:43:33.924Z] TRAPNO=000000000000000D CPU=5000000000000000 FAULTVADDR=000000012D925000
[2024-05-14T02:43:33.924Z] XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.924Z] XMM1=00007fe834a34438 (f: 883115072.000000, d: 6.948307e-310)
[2024-05-14T02:43:33.924Z] XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
[2024-05-14T02:43:33.924Z] XMM3=3e209caed7118cc7 (f: 3608251648.000000, d: 1.933896e-09)
[2024-05-14T02:43:33.924Z] XMM4=3ebb15b85180d120 (f: 1367396608.000000, d: 1.614382e-06)
[2024-05-14T02:43:33.924Z] XMM5=0000000000220000 (f: 2228224.000000, d: 1.100889e-317)
[2024-05-14T02:43:33.924Z] XMM6=3fe3c7c7fff73206 (f: 4294390272.000000, d: 6.181374e-01)
[2024-05-14T02:43:33.925Z] XMM7=402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
[2024-05-14T02:43:33.925Z] XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-14T02:43:33.925Z] Module=/Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-05-14T02:43:33.925Z] Module_base_address=0000000002134000 Symbol=walkStackFrames
[2024-05-14T02:43:33.925Z] Symbol_address=00000000021AE620
[2024-05-14T02:43:33.925Z] Target=2_90_20240514_65 (Mac OS X 13.2.1)
[2024-05-14T02:43:33.925Z] CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
[2024-05-14T02:43:33.925Z] ----------- Stack Backtrace -----------
[2024-05-14T02:43:33.925Z] walkStackFrames+0xb7d (0x00000000021AF19D [libj9vm29.dylib+0x7b19d])
[2024-05-14T02:43:33.925Z] getStackTraceForThread+0x20d (0x00000000027AE9AD [libjclse29.dylib+0x89ad])
[2024-05-14T02:43:33.925Z] Java_java_lang_Thread_getStackTraceImpl+0x51 (0x00000000027E82B1 [libjclse29.dylib+0x422b1])
[2024-05-14T02:43:33.925Z] ---------------------------------------
[2024-05-14T02:43:33.925Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/13 22:41:18 - please wait.

[2024-05-14T02:43:33.925Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
[2024-05-14T02:43:33.925Z] --------------------------------------------------
[2024-05-14T02:44:47.990Z] Test results: passed: 927; failed: 1
[2024-05-14T02:44:56.309Z] Report written to /Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/aqa-tests/TKG/output_17156542216869/jdk_lang_1/report/html/report.html
[2024-05-14T02:44:56.309Z] Results written to /Users/jenkins/workspace/Test_openjdk22_j9_sanity.openjdk_x86-64_mac_testList_1/aqa-tests/TKG/output_17156542216869/jdk_lang_1/work
[2024-05-14T02:44:56.309Z] Error: Some tests failed or other problems occurred.
[2024-05-14T02:44:56.309Z] -----------------------------------
[2024-05-14T02:44:56.309Z] jdk_lang_1_FAILED
JasonFengJ9 commented 5 months ago

JDK23 x86-64_mac(macx64rt7)

18:41:29  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
18:41:29  JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops -Xverbosegclog 

18:47:34  TEST: java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

18:47:34  Unhandled exception
18:47:34  Type=Segmentation error vmState=0x00000000
18:47:34  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
18:47:34  Handler1=0000000002531880 Handler2=0000000002286650
18:47:34  RDI=00007FEBC904DEF8 RSI=00000000049BE6AA RAX=B3894C0000010080 RBX=000070000DACB8C0
18:47:34  RCX=00000000148C0100 RDX=00007FEBC905FD48 R8=0000000000006085 R9=00007FEBC904E000
18:47:34  R10=00007FEBC90081F8 R11=00007FEBC9000000 R12=00007FEBC905FD48 R13=000070000DACB8C0
18:47:34  R14=89481A7703F88348 R15=000070000DACB920
18:47:34  RIP=000000000257419C GS=0000 FS=0000 RSP=000070000DACB820
18:47:34  RFlags=0000000000010282 CS=002B RBP=000070000DACB8A0 ERR=0000000000000000
18:47:34  TRAPNO=000000000000000D CPU=0000000000000000 FAULTVADDR=0000000000000000
18:47:34  XMM0=0000000100000001 (f: 1.000000, d: 2.121996e-314)
18:47:34  XMM1=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
18:47:34  XMM3=3dc5555588888888 (f: 2290649344.000000, d: 3.880511e-11)
18:47:34  XMM4=3e80000020000000 (f: 536870912.000000, d: 1.192093e-07)
18:47:34  XMM5=0000000000200000 (f: 2097152.000000, d: 1.036131e-317)
18:47:34  XMM6=3fda484090e5bb0a (f: 2430974720.000000, d: 4.106599e-01)
18:47:34  XMM7=402d1cb7eea86c0a (f: 4004015104.000000, d: 1.455609e+01)
18:47:34  XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:47:34  Module=/Users/jenkins/workspace/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
18:47:34  Module_base_address=00000000024F9000 Symbol=walkStackFrames
18:47:34  Symbol_address=0000000002573620
18:47:34  Target=2_90_20240528_170 (Mac OS X 14.2.1)
18:47:34  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
18:47:34  ----------- Stack Backtrace -----------
18:47:34  walkStackFrames+0xb7d (0x000000000257419D [libj9vm29.dylib+0x7b19d])
18:47:34  getStackTraceForThread+0x20d (0x0000000002B30D1D [libjclse29.dylib+0x9d1d])
18:47:34  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x0000000002B6A651 [libjclse29.dylib+0x43651])
18:47:34  ---------------------------------------
18:47:34  JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/28 18:45:34 - please wait.

18:47:34  TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
18:47:34  --------------------------------------------------
18:48:58  Test results: passed: 925; failed: 1
18:49:05  Report written to /Users/jenkins/workspace/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal_testList_1/aqa-tests/TKG/output_17169360886410/jdk_lang_1/report/html/report.html
18:49:05  Results written to /Users/jenkins/workspace/Test_openjdknext_j9_sanity.openjdk_x86-64_mac_Personal_testList_1/aqa-tests/TKG/output_17169360886410/jdk_lang_1/work
18:49:05  Error: Some tests failed or other problems occurred.
18:49:05  -----------------------------------
18:49:05  jdk_lang_1_FAILED
pshipton commented 4 months ago

This has been deferred before, deferring it again.

pshipton commented 3 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OMR_testList_0/103 jdk_lang_0 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OMR_testList_0/103/openjdk_test_output.tar.gz

16:34:07  2024-08-01T20:32:06.533566Z 41256 remaining ...
16:34:07  2024-08-01T20:32:07.034658Z 38839 remaining ...
16:34:07  STDERR:
16:34:07  Unhandled exception
16:34:07  Type=Segmentation error vmState=0x00000000
16:34:07  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
16:34:07  Handler1=0000000010E392A0 Handler2=00000000111426D0 InaccessibleAddress=0000000000000009
16:34:07  RDI=00007F8F6EEB3048 RSI=00000000A93B144D RAX=0000000000000008 RBX=0000700010C3B8C0
16:34:07  RCX=00000000148C0100 RDX=0000000000000001 R8=000000000000AEDA R9=000000000000AEFF
16:34:07  R10=0000000080000000 R11=00007F8F6EEB0830 R12=000000001D01A288 R13=00000000140C0100
16:34:07  R14=0000000000000006 R15=0000700010C3B920
16:34:07  RIP=0000000010E7B2E9 GS=0000 FS=0000 RSP=0000700010C3B820
16:34:07  RFlags=0000000000010203 CS=002B RBP=0000700010C3B8A0 ERR=0000000900000004
16:34:07  TRAPNO=000000040000000E CPU=0009000000040000 FAULTVADDR=0000000000000009
16:34:07  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM1=000000001cfcf678 (f: 486340224.000000, d: 2.402840e-315)
16:34:07  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
16:34:07  XMM3=3da07f1df5af0701 (f: 4121888512.000000, d: 7.501763e-12)
16:34:07  XMM4=3e6565c23f1987b7 (f: 1058637760.000000, d: 3.985594e-08)
16:34:07  XMM5=00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
16:34:07  XMM6=3fe5bd7d30e71c73 (f: 820452480.000000, d: 6.793810e-01)
16:34:07  XMM7=402d1cb7eea86c0a (f: 4004015104.000000, d: 1.455609e+01)
16:34:07  XMM8=0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
16:34:07  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
16:34:07  Module=/Users/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
16:34:07  Module_base_address=0000000010E00000 Symbol=walkStackFrames
16:34:07  Symbol_address=0000000010E7ABA0
16:34:07  Target=2_90_20240801_123 (Mac OS X 10.15.7)
16:34:07  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
16:34:07  ----------- Stack Backtrace -----------
16:34:07  walkStackFrames+0x74a (0x0000000010E7B2EA [libj9vm29.dylib+0x7b2ea])
16:34:07  getStackTraceForThread+0x20d (0x00000000135710ED [libjclse29.dylib+0x90ed])
16:34:07  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x00000000135AB211 [libjclse29.dylib+0x43211])
16:34:07  ---------------------------------------
pshipton commented 3 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/1

00:07:51  Type=Segmentation error vmState=0x00000000
00:07:51  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
00:07:51  Handler1=00000000094454D0 Handler2=0000000009199580
00:07:51  RDI=00007FC319A10FA8 RSI=000000000B8DB48A RAX=B3894C0000010080 RBX=0000700010B618C0
00:07:51  RCX=00000000148C0100 RDX=000000000CF9F828 R8=00000000000000CA R9=00000000FFE00000
00:07:51  R10=000000000FFFFFFF R11=0000000000000000 R12=000000000CF9F828 R13=0000700010B618C0
00:07:51  R14=89481A7703F88348 R15=0000700010B61920
00:07:51  RIP=0000000009487E4C GS=0000 FS=0000 RSP=0000700010B61820
00:07:51  RFlags=0000000000010282 CS=002B RBP=0000700010B618A0 ERR=0000000000000000
00:07:51  TRAPNO=000000000000000D CPU=0000000000000000 FAULTVADDR=0000000000000000
00:07:51  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM1=0800000008000000 (f: 134217728.000000, d: 3.785767e-270)
00:07:51  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
00:07:51  XMM3=3dfa5e06980edf72 (f: 2551111424.000000, d: 3.836945e-10)
00:07:51  XMM4=3ea26d58d6db807c (f: 3604709376.000000, d: 5.491715e-07)
00:07:51  XMM5=00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
00:07:51  XMM6=3fbd4313d66cb35d (f: 3597447936.000000, d: 1.143048e-01)
00:07:51  XMM7=402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
00:07:51  XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:07:51  Module=/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
00:07:51  Module_base_address=000000000940C000 Symbol=walkStackFrames
00:07:51  Symbol_address=00000000094872D0
00:07:51  Target=2_90_20240802_1 (Mac OS X 14.3)
00:07:51  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
00:07:51  ----------- Stack Backtrace -----------
00:07:51  walkStackFrames+0xb7d (0x0000000009487E4D [libj9vm29.dylib+0x7be4d])
00:07:51  getStackTraceForThread+0x20d (0x0000000009A41D5D [libjclse29.dylib+0x8d5d])
00:07:51  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x0000000009A7BE81 [libjclse29.dylib+0x42e81])
JasonFengJ9 commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/5/consoleFull

02:56:36  openjdk version "23-internal" 2024-09-17
02:56:36  OpenJDK Runtime Environment (build 23-internal-adhoc.jenkins.BuildJDK23x86-64macNightly)
02:56:36  Eclipse OpenJ9 VM (build master-1d2f429d4ce, JRE 23 Mac OS X amd64-64-Bit Compressed References 20240808_5 (JIT enabled, AOT enabled)
02:56:36  OpenJ9   - 1d2f429d4ce
02:56:36  OMR      - 9ccff3cf2b8
02:56:36  JCL      - e799a69f73f based on jdk-23+35)
02:56:36  
02:56:36  ===============================================
02:56:36  Running test jdk_lang_1 ...
02:56:36  ===============================================
02:56:36  jdk_lang_1 Start Time: Fri Aug  9 02:56:35 2024 Epoch Time (ms): 1723186595931
02:56:36  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
02:56:36  JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops -Xverbosegclog 

03:03:41  TEST: java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

03:03:41  
03:03:41  Unhandled exception
03:03:41  Type=Segmentation error vmState=0x00000000
03:03:41  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
03:03:41  Handler1=000000000A7784D0 Handler2=000000000A4CC580
03:03:41  RDI=00007FF2FDA10B60 RSI=00000000389F278C RAX=0000050C8B480000 RBX=000070000D8548C0
03:03:41  RCX=0000000000000000 RDX=7FF3006ADEC8B848 R8=0000000000000C7A R9=0000000000000CB0
03:03:41  R10=00000000FFFF0000 R11=000000000000005D R12=00007FF300836010 R13=00000000140C0100
03:03:41  R14=800CFF952BD3FD0F R15=000000000A3F6008
03:03:41  RIP=000000000A7BABE1 GS=0000 FS=0000 RSP=000070000D854820
03:03:41  RFlags=0000000000010297 CS=002B RBP=000070000D8548A0 ERR=6E60D00000000000
03:03:41  TRAPNO=000000000000000D CPU=D000000000000000 FAULTVADDR=00007FEE6E60D000
03:03:41  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM1=00007ff2ff1aa728 (f: 4279936768.000000, d: 6.950596e-310)
03:03:41  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
03:03:41  XMM3=3e39cc73ad89ddb0 (f: 2911493632.000000, d: 6.006714e-09)
03:03:41  XMM4=3eccd4455d92f3e1 (f: 1569911808.000000, d: 3.436706e-06)
03:03:41  XMM5=0000000000200000 (f: 2097152.000000, d: 1.036131e-317)
03:03:41  XMM6=3fd7bede0a37afc0 (f: 171421632.000000, d: 3.710246e-01)
03:03:41  XMM7=402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
03:03:41  XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
03:03:41  Module=/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
03:03:41  Module_base_address=000000000A73F000 Symbol=walkStackFrames
03:03:41  Symbol_address=000000000A7BA2D0
03:03:41  Target=2_90_20240808_5 (Mac OS X 12.1)
03:03:41  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
03:03:41  ----------- Stack Backtrace -----------
03:03:41  walkStackFrames+0x912 (0x000000000A7BABE2 [libj9vm29.dylib+0x7bbe2])
03:03:41  getStackTraceForThread+0x20d (0x000000000AD75D5D [libjclse29.dylib+0x8d5d])
03:03:41  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000ADAFE81 [libjclse29.dylib+0x42e81])
03:03:41  ---------------------------------------
03:03:41  JVMDUMP039I Processing dump event "gpf", detail "" at 2024/08/09 03:01:37 - please wait.
pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Release_testList_0/2 - mac12-x86-1a\ jdk_lang_0 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Release_testList_0/2/openjdk_test_output.tar.gz

11:23:45  Type=Segmentation error vmState=0x00000000
11:23:45  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
11:23:45  Handler1=0000000008AE81A0 Handler2=000000000883D580
11:23:45  RDI=00007FF1F670DF60 RSI=00000000241042CD RAX=480000083B840FC0 RBX=000070000D22D8C0
11:23:45  RCX=0000000000000000 RDX=3B45C08B45C08B4C R8=00000000000009C9 R9=00000000000009F0
11:23:45  R10=00000000FFFF0000 R11=00000000000000C2 R12=0000000017781520 R13=00000000140C0100
11:23:45  R14=C4BA3F74C313285F R15=0000000008767008
11:23:45  RIP=0000000008B2AB31 GS=0000 FS=0000 RSP=000070000D22D820
11:23:45  RFlags=0000000000010297 CS=002B RBP=000070000D22D8A0 ERR=214A900000000000
11:23:45  TRAPNO=000000000000000D CPU=9000000000000000 FAULTVADDR=00000001214A9000
11:23:45  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM1=000000001773e948 (f: 393472320.000000, d: 1.944012e-315)
11:23:45  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
11:23:45  XMM3=3e33347c20c0b5c8 (f: 549500352.000000, d: 4.471517e-09)
11:23:45  XMM4=3ec7ae05bdda4cf4 (f: 3185200384.000000, d: 2.822849e-06)
11:23:45  XMM5=00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
11:23:45  XMM6=3fde4c71a8687704 (f: 2825418496.000000, d: 4.734158e-01)
11:23:45  XMM7=402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
11:23:45  XMM8=0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
11:23:45  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
11:23:45  Module=/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
11:23:45  Module_base_address=0000000008AAF000 Symbol=walkStackFrames
11:23:45  Symbol_address=0000000008B2A220
11:23:45  Target=2_90_20240811_2 (Mac OS X 12.1)
11:23:45  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
11:23:45  ----------- Stack Backtrace -----------
11:23:45  walkStackFrames+0x912 (0x0000000008B2AB32 [libj9vm29.dylib+0x7bb32])
11:23:45  getStackTraceForThread+0x20d (0x00000000090E4F1D [libjclse29.dylib+0x8f1d])
11:23:45  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000911EE81 [libjclse29.dylib+0x42e81])
pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/224 - mac10-x86-2 jdk_lang_j9_0 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/224/openjdk_test_output.tar.gz 19:17:21 23:15:46.624 0x7f8ea89aa100 j9vm.224 * ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK21_x86-64_mac_Nightly/openj9/runtime/vm/swalk.c:116: ((sp >= stackStart))

pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/16/ - mac14-x86-3a jdk_lang_0 java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/16/openjdk_test_output.tar.gz

05:00:54  2024-08-28T08:58:33.162077Z 427080 remaining ...
05:00:54  STDERR:
05:00:54  WARNING: A restricted method in java.lang.foreign.SymbolLookup has been called
05:00:54  WARNING: java.lang.foreign.SymbolLookup::libraryLookup has been called by jdk.test.lib.thread.VThreadPinner in an unnamed module
05:00:54  WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
05:00:54  WARNING: Restricted methods will be blocked in a future release unless native access is enabled
05:00:54  
05:00:54  08:58:33.457 0x132a5100    j9vm.224    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK23_x86-64_mac_Nightly/openj9/runtime/vm/swalk.c:117: ((sp <= endOfStack))
pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_1/26/ - mac10-x86-1

13:02:40  Type=Segmentation error vmState=0x00000000
13:02:40  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
13:02:40  Handler1=00000000052392A0 Handler2=00000000039054C0
13:02:40  RDI=00007FCBF8C480E0 RSI=0000000031D49C76 RAX=2C247C83C88B4D40 RBX=000070000E0418C0
13:02:40  RCX=0000000000000000 RDX=4D2C244489C38B4C R8=00000000000043E4 R9=0000000000004408
13:02:40  R10=00000000FFFFFF00 R11=00007FCBF8C47250 R12=00007FCBFE0C2EA0 R13=00000000140C0100
13:02:40  R14=B2D3DBBB7B853B8F R15=0000000003891228
13:02:40  RIP=000000000527B771 GS=0000 FS=0000 RSP=000070000E041820
13:02:40  RFlags=0000000000010293 CS=002B RBP=000070000E0418A0 ERR=8C765E1400000000
13:02:40  TRAPNO=000000000000000D CPU=5E14000000000000 FAULTVADDR=00007FBD8C765E14
13:02:40  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM1=00007fcc0085e288 (f: 8774280.000000, d: 6.942322e-310)
13:02:40  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
13:02:40  XMM3=3e12577f5ab6d382 (f: 1521931136.000000, d: 1.067633e-09)
13:02:40  XMM4=3eb23a264c8066d2 (f: 1283483392.000000, d: 1.086423e-06)
13:02:40  XMM5=00000000002e0000 (f: 3014656.000000, d: 1.489438e-317)
13:02:40  XMM6=3fe3ea33936b2f5c (f: 2473275136.000000, d: 6.223390e-01)
13:02:40  XMM7=402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
13:02:40  XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
13:02:40  Module=/Users/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
13:02:40  Module_base_address=0000000005200000 Symbol=walkStackFrames
13:02:40  Symbol_address=000000000527AE60
13:02:40  Target=2_90_20240828_37 (Mac OS X 10.15.7)
13:02:40  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
13:02:40  ----------- Stack Backtrace -----------
13:02:40  walkStackFrames+0x912 (0x000000000527B772 [libj9vm29.dylib+0x7b772])
pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/19

09:41:46  Type=Segmentation error vmState=0x00000000
09:41:46  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
09:41:46  Handler1=000000000A7EB4D0 Handler2=000000000A53F580
09:41:46  RDI=00007FD1C4C10528 RSI=000000000CC82DEA RAX=B3894C0000010080 RBX=00007000028188C0
09:41:46  RCX=00000000148C0100 RDX=000000000E349638 R8=0000000000000C13 R9=00000000000000C0
09:41:46  R10=00007FD1C4C00000 R11=00000000FFFFE000 R12=000000000E349638 R13=00007000028188C0
09:41:46  R14=89481A7703F88348 R15=0000700002818920
09:41:46  RIP=000000000A82DE4C GS=0000 FS=0000 RSP=0000700002818820
09:41:46  RFlags=0000000000010282 CS=002B RBP=00007000028188A0 ERR=0F4D100800000000
09:41:46  TRAPNO=000000000000000D CPU=1008000000000000 FAULTVADDR=000000010F4D1008
09:41:46  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM1=000000000e1ce9f8 (f: 236775936.000000, d: 1.169829e-315)
09:41:46  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
09:41:46  XMM3=3e4d894764288cf1 (f: 1680379136.000000, d: 1.375389e-08)
09:41:46  XMM4=3ed90aa44e9025b7 (f: 1318069632.000000, d: 5.970375e-06)
09:41:46  XMM5=00000000001e0000 (f: 1966080.000000, d: 9.713726e-318)
09:41:46  XMM6=3fbc5e548f5bc743 (f: 2405156608.000000, d: 1.108144e-01)
09:41:46  XMM7=402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
09:41:46  XMM8=0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
09:41:46  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
09:41:46  Module=/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
09:41:46  Module_base_address=000000000A7B2000 Symbol=walkStackFrames
09:41:46  Symbol_address=000000000A82D2D0
09:41:46  Target=2_90_20240830_21 (Mac OS X 13.4.1)
09:41:46  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
09:41:46  ----------- Stack Backtrace -----------
09:41:46  walkStackFrames+0xb7d (0x000000000A82DE4D [libj9vm29.dylib+0x7be4d])
09:41:46  getStackTraceForThread+0x20d (0x000000000ADE7D5D [libjclse29.dylib+0x8d5d])
09:41:46  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000AE21E81 [libjclse29.dylib+0x42e81])
pshipton commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_1/27 https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_0/27/ - mac12-x86-1a

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_0/27/openjdk_test_output.tar.gz

06:00:09  Type=Segmentation error vmState=0x00000000
06:00:09  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
06:00:09  Handler1=000000000359B2A0 Handler2=00000000032EF4C0
06:00:09  RDI=00007FC767306178 RSI=0000000033505484 RAX=000001AC850F0020 RBX=000070000661A8C0
06:00:09  RCX=0000000000000000 RDX=247C830000441F0F R8=00000000000001EC R9=0000000000000212
06:00:09  R10=00000000FFFC0000 R11=0000000000000367 R12=00007FC76481BAB0 R13=00000000140C0100
06:00:09  R14=DB837D00033AC2B8 R15=0000000003219228
06:00:09  RIP=00000000035DD4B1 GS=0000 FS=0000 RSP=000070000661A820
06:00:09  RFlags=0000000000010297 CS=002B RBP=000070000661A8A0 ERR=034A500000000000
06:00:09  TRAPNO=000000000000000D CPU=5000000000000000 FAULTVADDR=00000000034A5000
06:00:09  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM1=00007fc76186d7a8 (f: 1636226944.000000, d: 6.941341e-310)
06:00:09  XMM2=00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
06:00:09  XMM3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM5=4530000000000000 (f: 0.000000, d: 1.934281e+25)
06:00:09  XMM6=3fd9f323ecbf984c (f: 3971979264.000000, d: 4.054651e-01)
06:00:09  XMM7=402791272ee9dd8e (f: 787078528.000000, d: 1.178350e+01)
06:00:09  XMM8=0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
06:00:09  XMM9=408b600000000000 (f: 0.000000, d: 8.760000e+02)
06:00:09  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:00:09  Module=/Users/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_OpenJDK21_testList_0/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
06:00:09  Module_base_address=0000000003562000 Symbol=walkStackFrames
06:00:09  Symbol_address=00000000035DCBA0
06:00:09  Target=2_90_20240905_38 (Mac OS X 12.1)
06:00:09  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
06:00:09  ----------- Stack Backtrace -----------
06:00:09  walkStackFrames+0x912 (0x00000000035DD4B2 [libj9vm29.dylib+0x7b4b2])
06:00:09  getStackTraceForThread+0x20d (0x0000000003B8CD4D [libjclse29.dylib+0x8d4d])
06:00:09  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x0000000003BC6E71 [libjclse29.dylib+0x42e71])
pshipton commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/236

JasonFengJ9 commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/24/consoleFull

03:56:41  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
03:56:41  JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops -Xverbosegclog 

04:05:52  TEST: java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0

04:05:52  Unhandled exception
04:05:52  Type=Segmentation error vmState=0x00000000
04:05:52  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
04:05:52  Handler1=000000000A1004C0 Handler2=0000000009E54580
04:05:52  RDI=00007FC049C0C5A8 RSI=000000000C596D4A RAX=B3894C0000010080 RBX=0000700003BE28C0
04:05:52  RCX=00000000148C0100 RDX=00007FC04B8E3D08 R8=0000000000000082 R9=00000000FFE00000
04:05:52  R10=0000000000000000 R11=0000000000000000 R12=00007FC04B8E3D08 R13=0000700003BE28C0
04:05:52  R14=89481A7703F88348 R15=0000700003BE2920
04:05:52  RIP=000000000A142E3C GS=0000 FS=0000 RSP=0000700003BE2820
04:05:52  RFlags=0000000000010282 CS=002B RBP=0000700003BE28A0 ERR=0000000000000000
04:05:52  TRAPNO=000000000000000D CPU=0000000000000000 FAULTVADDR=0000000000000000
04:05:52  XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM1=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM2=000000005f000000 (f: 1593835520.000000, d: 7.874594e-315)
04:05:52  XMM3=3e245a6d37f9359d (f: 939079040.000000, d: 2.369428e-09)
04:05:52  XMM4=3ebf03227caa343f (f: 2091529216.000000, d: 1.848474e-06)
04:05:52  XMM5=0000000000290000 (f: 2686976.000000, d: 1.327543e-317)
04:05:52  XMM6=3fe20cdcd192ab6e (f: 3516050176.000000, d: 5.640701e-01)
04:05:52  XMM7=402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
04:05:52  XMM8=0000000040400000 (f: 1077936128.000000, d: 5.325712e-315)
04:05:52  XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:05:52  Module=/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
04:05:52  Module_base_address=000000000A0C7000 Symbol=walkStackFrames
04:05:52  Symbol_address=000000000A1422C0
04:05:52  Target=2_90_20240909_27 (Mac OS X 14.3)
04:05:52  CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
04:05:52  ----------- Stack Backtrace -----------
04:05:52  walkStackFrames+0xb7d (0x000000000A142E3D [libj9vm29.dylib+0x7be3d])
04:05:52  getStackTraceForThread+0x20d (0x000000000A6FDD5D [libjclse29.dylib+0x8d5d])
04:05:52  Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000A737E81 [libjclse29.dylib+0x42e81])
04:05:52  ---------------------------------------
04:05:52  JVMDUMP039I Processing dump event "gpf", detail "" at 2024/09/10 05:03:36 - please wait.

04:05:52  TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
04:05:52  --------------------------------------------------
04:07:42  Test results: passed: 928; failed: 1
04:07:44  Report written to /Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_17259550007658/jdk_lang_1/report/html/report.html
04:07:44  Results written to /Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_17259550007658/jdk_lang_1/work
04:07:44  Error: Some tests failed or other problems occurred.
04:07:44  -----------------------------------
04:07:44  jdk_lang_1_FAILED
a7ehuo commented 1 month ago

I looked at the last reported crash. I got some backtrace in lldb but it's missing symbols from getStackTraceForThread and Java_java_lang_Thread_getStackTraceImpl.

The backtrace from the console 

----------- Stack Backtrace -----------
walkStackFrames+0xb7d (0x000000000A142E3D [libj9vm29.dylib+0x7be3d])
getStackTraceForThread+0x20d (0x000000000A6FDD5D [libjclse29.dylib+0x8d5d])
Java_java_lang_Thread_getStackTraceImpl+0x51 (0x000000000A737E81 [libjclse29.dylib+0x42e81])
---------------------------------------

The backtrace from lldb 

(lldb) bt
* thread #31
  * frame #0: 0x00007ff818a967a2 libsystem_kernel.dylib`__wait4 + 10
    frame #1: 0x0000000009e519de libj9prt29.dylib`omrdump_create(portLibrary=<unavailable>, filename="/Users/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_17259550007658/jdk_lang_1/work/scratch/0/core.20240910.050336.76789.0001.dmp", dumpType=<unavailable>, userData=<unavailable>) at omrosdump.c:679:3 [opt]
    frame #2: 0x0000000009f37808
    frame #3: 0x0000000009f3c905
    frame #4: 0x0000000009e53072 libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000009d7e008, fn=<unavailable>, fn_arg=<unavailable>, handler=<unavailable>, handler_arg=0x0000000000000000, flags=<unavailable>, result=0x0000700003be04c8) at omrsignal.c:425:12 [opt]
    frame #5: 0x0000000009f3c53b
    frame #6: 0x0000000009f559a4
    frame #7: 0x000000000a101553 libj9vm29.dylib`generateDiagnosticFiles(portLibrary=<unavailable>, userData=<unavailable>) at gphandle.c:1162:3 [opt]
    frame #8: 0x0000000009e53072 libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000009d7e008, fn=<unavailable>, fn_arg=0x0000700003be1330, handler=<unavailable>, handler_arg=0x0000700003be1390, flags=<unavailable>, result=0x0000700003be1328) at omrsignal.c:425:12 [opt]
    frame #9: 0x000000000a1007e8 libj9vm29.dylib`vmSignalHandler(portLibrary=0x0000000009d7e008, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:837:2 [opt]
    frame #10: 0x000000000a100503 libj9vm29.dylib`structuredSignalHandler(portLibrary=<unavailable>, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:634:9 [opt]
    frame #11: 0x0000000009e54702 libj9prt29.dylib`mainSynchSignalHandler(signal=<unavailable>, sigInfo=<unavailable>, contextInfo=<unavailable>) at omrsignal.c:1066:14 [opt]
    frame #12: 0x00007ff818afe37d libsystem_platform.dylib`_sigtramp + 29
    frame #13: 0x000000000a142e3d libj9vm29.dylib`walkStackFrames at swalk.c:748:25 [opt]
    frame #14: 0x000000000a142d44 libj9vm29.dylib`walkStackFrames(currentThread=<unavailable>, walkState=<unavailable>) at swalk.c:335:5 [opt]
    frame #15: 0x000000000a6fdd5d // console log: getStackTraceForThread+0x20d
    frame #16: 0x000000000a737e81 // console log: Java_java_lang_Thread_getStackTraceImpl+0x51
    frame #17: 0x00000000374b896f
    frame #18: 0x000000000a0dc0d7 libj9vm29.dylib`::runJavaThread(currentThread=<unavailable>) at callin.cpp:682:4 [opt]
    frame #19: 0x000000000a15435b libj9vm29.dylib`javaProtectedThreadProc(portLibrary=<unavailable>, entryarg=0x00007fc04c157700) at vmthread.cpp:2128:3 [opt]
    frame #20: 0x0000000009e53072 libj9prt29.dylib`omrsig_protect(portLibrary=0x0000000009d7e008, fn=<unavailable>, fn_arg=0x00007fc04c157700, handler=<unavailable>, handler_arg=0x00007fc04c157700, flags=<unavailable>, result=0x0000700003be2f70) at omrsignal.c:425:12 [opt]
    frame #21: 0x000000000a154298 libj9vm29.dylib`::javaThreadProc(entryarg=0x00007fc04b812820) at vmthread.cpp:405:2 [opt]
    frame #22: 0x0000000009db26c0
    frame #23: 0x00007ff818ad0202 libsystem_pthread.dylib`_pthread_start + 99
    frame #24: 0x00007ff818acbbab libsystem_pthread.dylib`thread_start + 15

It crashes in walkMethodFrame. The walkState->method (0x000000000c596d4a) looks invalid to me [1].

The walkState->currentThread (0x00007fc04c157700) is different from the walkState->walkThread (0x00007fc04b8a8d00) or targetThread. Both threads involve virtual thread methods [2]. The getStackTraceImpl on the currentThread is triggered by VirtualThread.asyncGetStackTrace. The top method on the walkThread stack is VirtualThread.unmount which is inlined in VirtualThread.runContinuation.

I looked at how walkState->method is set up in walkMethodTypeFrame and tried to rebuild J9SFMethodFrame from walkState->walkSP, but the data doesn't match walkState [3].

[1]

(lldb) fr select 13
frame #13: 0x000000000a142e3d libj9vm29.dylib`walkStackFrames at swalk.c:748:25 [opt]
   745          J9ROMMethod * romMethod = J9_ROM_METHOD_FROM_RAM_METHOD(walkState->method);
   746  
   747          walkState->constantPool = UNTAGGED_METHOD_CP(walkState->method);
-> 748          walkState->argCount = J9_ARG_COUNT_FROM_ROM_METHOD(romMethod);
   749  
   750          if (walkState->flags & J9_STACKWALK_ITERATE_O_SLOTS) {
   751              WALK_METHOD_CLASS(walkState);
(lldb) 
(lldb) p *walkState
(J9StackWalkState) $1 = {
  previous = NULL
  walkThread = 0x00007fc04b8a8d00
  javaVM = 0x00007fc04b812820
  flags = 344719616
  bp = 0x00007fc04b8e3d28
  unwindSP = 0x00007fc04b8e3d08
  pc = 0x0000000000000003 ""
  nextPC = 0x0000000000000000
  sp = 0x00007fc04b8e3d08
  arg0EA = 0x00007fc04b8e3d30
  literals = NULL
  walkSP = 0x00007fc04b8e3d08
  argCount = 0
  constantPool = 0xb3894c0000010080
  method = 0x000000000c596d4a
  jitInfo = NULL
  frameFlags = 229650784
  resolveFrameFlags = 0
  skipCount = 0
  maxFrames = 0
  userData1 = 0x0000000000000000
  userData2 = 0x0000000000000000
  userData3 = 0x0000000000000000
  userData4 = 0x0000000000000000
  framesWalked = 0
  frameWalkFunction = 0x0000000000000000
  objectSlotWalkFunction = 0x0000000000000000
  returnAddressWalkFunction = 0x0000000000000000
  cache = 0x00007fc049c0c290
  restartPoint = 0x0000000000000000
  restartException = 0x0000000000000000
  inlinerMap = 0x0000000000000000
  inlineDepth = 0
  cacheCursor = 0x00007fc049c0c290
  decompilationRecord = NULL
  registerEAs = {
    jit_rax = 0x0000000000000000
    jit_rbx = 0x0000000000000000
    jit_rcx = 0x0000000000000000
    jit_rdx = 0x0000000000000000
    jit_rdi = 0x0000000000000000
    jit_rsi = 0x0000000000000000
    jit_rbp = 0x0000000000000000
    jit_rsp = 0x0000000000000000
    jit_r8 = 0x0000000000000000
    jit_r9 = 0x0000000000000000
    jit_r10 = 0x0000000000000000
    jit_r11 = 0x0000000000000000
    jit_r12 = 0x0000000000000000
    jit_r13 = 0x0000000000000000
    jit_r14 = 0x0000000000000000
    jit_r15 = 0x0000000000000000
  }
  walkedEntryLocalStorage = 0x0000700003ef4d58
  i2jState = 0x0000700003ef4d68
  decompilationStack = NULL
  pcAddress = 0x00007fc04b8a8d28
  outgoingArgCount = 0
  objectSlotBitVector = 0x0000000000000000
  elsBitVector = 0
  savedObjectSlotWalkFunction = 0x0000000000000000
  bytecodePCOffset = -1
  dropToCurrentFrame = 0x000000000a143540 (libj9vm29.dylib`dropToCurrentFrame at swalk.c:1601)
  j2iFrame = 0x0000000000000000
  previousFrameFlags = 0
  slotIndex = 0
  slotType = 0
  currentThread = 0x00007fc04c157700
  linearSlotWalker = 0x0000000000000000
  inlinedCallSite = 0x0000000000000000
  stackMap = 0x0000000000000000
  inlineMap = 0x0000000000000000
  loopBreaker = 0
}
> !j9method 0x000000000c596d4a
J9Method at 0xc596d4a {
  Fields for J9Method:
    0x0: U8* bytecodes = !j9x 0x89481A7703F88348
    0x8: struct J9ConstantPool* constantPool = !j9constantpool 0xB3894C0000010080 (flags = 0x3)
    0x10: void* methodRunAddress = !j9x 0xAB058D4800000108
    0x18: volatile void* extra = !j9x 0xC35D5E415BFFFF97
}
Signature: <FAULT>
ROM Method: !j9rommethod 0x89481A7703F88334
Next Method: !j9method 0x000000000C596D6A

[2] === currentThread (0x00007fc04c157700) == 

> !stack 0x00007fc04c157700 // currentThread
<7fc04c157700>  !j9method 0x00007FC04E785D30   java/lang/Thread.getStackTraceImpl()Ljava/lang/Throwable; // Native implementation Java_java_lang_Thread_getStackTraceImpl 
<7fc04c157700>  !j9method 0x00007FC04E785D10   java/lang/Thread.getStackTrace0()Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC04E785CF0   java/lang/Thread.asyncGetStackTrace()[Ljava/lang/StackTraceElement;
<7fc04c157700>  !j9method 0x00007FC04E7A4148   java/lang/VirtualThread.asyncGetStackTrace()[Ljava/lang/StackTraceElement;
<7fc04c157700>  !j9method 0x00007FC04E785CD0   java/lang/Thread.getStackTrace()[Ljava/lang/StackTraceElement;
<7fc04c157700>  !j9method 0x00007FC0503E9BE8   GetStackTraceALotWhenPinned.main([Ljava/lang/String;)V
<7fc04c157700>  !j9method 0x00007FC04A07CDA8   java/lang/invoke/LambdaForm$DMH/0x000000004a07c420.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)V
<7fc04c157700>  !j9method 0x00007FC04B8BA1B8   java/lang/invoke/LambdaForm$MH/0x000000004b8b9820.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC04B8BADB8   java/lang/invoke/LambdaForm$MH/0x000000004b8ba420.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC05035F4A0   jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC05035F460   jdk/internal/reflect/DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC04E779D80   java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<7fc04c157700>  !j9method 0x00007FC0503E96F0   com/sun/javatest/regtest/agent/MainWrapper$MainTask.run()V
<7fc04c157700>  !j9method 0x00007FC04E785890   java/lang/Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V
<7fc04c157700>  !j9method 0x00007FC04E785870   java/lang/Thread.run()V
<7fc04c157700>                          JNI call-in frame
<7fc04c157700>                          Native method frame

=== walkThread (0x00007fc04b8a8d00) == 

> !stack 0x00007fc04b8a8d00 // walkThread 
<7fc04b8a8d00>                          known but unhandled frame type com.ibm.j9ddr.vm29.pointer.U8Pointer @ 0x00000005 

 FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT 

<7fc04b8a8d00>  !j9method 0x00007FC04E7A3DE8   java/lang/VirtualThread.unmount()V // JIT inlined
<7fc04b8a8d00>  !j9method 0x00007FC04E7A3D08   java/lang/VirtualThread.runContinuation()V // JIT
<7fc04b8a8d00>  !j9method 0x00007FC050664898   java/lang/VirtualThread$$Lambda/0x000000005062bb48.run()V
<7fc04b8a8d00>  !j9method 0x00007FC05066C5C8   java/util/concurrent/ForkJoinTask$AdaptedRunnableAction.exec()Z
<7fc04b8a8d00>  !j9method 0x00007FC05064D7A8   java/util/concurrent/ForkJoinTask.doExec()V
<7fc04b8a8d00>  !j9method 0x00007FC05064F640   java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(Ljava/util/concurrent/ForkJoinTask;I)V
<7fc04b8a8d00>  !j9method 0x00007FC050627E68   java/util/concurrent/ForkJoinPool.runWorker(Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7fc04b8a8d00>  !j9method 0x00007FC050629870   java/util/concurrent/ForkJoinWorkerThread.run()V
<7fc04b8a8d00>                          JNI call-in frame
<7fc04b8a8d00>                          Native method frame

[3]

(lldb) p *((J9SFMethodFrame *) ((U_8*) walkState->walkSP + (UDATA) walkState->literals))
(J9SFMethodFrame) $1 = {
  method = NULL
  specialFrameFlags = 11010048
  savedCP = NULL
  savedPC = 0x00000000371df153 "L\x8b\xc3H\x8b\x9c$\xc0"
  savedA0 = 0x00007fc04b8e3d32
}
pshipton commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/34 https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/35

hzongaro commented 1 month ago

I'm trying to capture some debugging information from earlier iterations of the stack walk in hopes of being able to reconstruct how we get to the point of failure. My first grinder run showed no failures. Trying again. . . .

hzongaro commented 1 month ago

I suspect that the thread whose stack is being walked might still be running, and that that is messing up the stack walking.

I added some debugging code into swalk.c to capture the values of walkState->walkThread->pc at various points before the while loop in walkStackFrames — you can find that in my branch investigate-issue18910 — and I ran that in internal grinder run 43760.

Using lldb to look at one of the core files captured in https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Grinder_iteration_3/3406/openjdk_test_output.tar.gz:

(lldb) target create jdk/bin/java -core aqa-tests/TKG/output_17278432338371/jdk_custom_0_ITER_3/work/java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned_id0/core.20241002.003157.42718.0001.dmp
(lldb) thread sel 31  
(lldb) fr sel 14
(lldb) bt 31
...
    frame #9: 0x00000000011596e8 libj9vm29.dylib`vmSignalHandler(portLibrary=0x0000000000dd7228, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:837:2 [opt]
    frame #10: 0x0000000001159403 libj9vm29.dylib`structuredSignalHandler(portLibrary=<unavailable>, gpType=<unavailable>, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:634:9 [opt]
    frame #11: 0x0000000000ead682 libj9prt29.dylib`mainSynchSignalHandler(signal=<unavailable>, sigInfo=<unavailable>, contextInfo=<unavailable>) at omrsignal.c:1066:14 [opt]
    frame #12: 0x00007ff813ea0c1d libsystem_platform.dylib`_sigtramp + 29
    frame #13: 0x000000000119b7bc libj9vm29.dylib`walkStackFrames at swalk.c:1058:25 [opt]
  * frame #14: 0x000000000119b790 libj9vm29.dylib`walkStackFrames(currentThread=<unavailable>, walkState=<unavailable>) at swalk.c:405:6 [opt]
    frame #15: 0x0000000001782cdd libjclse29.dylib`getStackTraceForThread(currentThread=0x0000000004b78f00, targetThread=0x0000000004c0ab00, skipCount=<unavailable>, threadObject=<unavailable>) at getstacktrace.c:94:8 [opt]
    frame #16: 0x00000000017bce01 libjclse29.dylib`Java_java_lang_Thread_getStackTraceImpl(env=0x0000000004b78f00, rcv=0x0000000004bd1810) at thread.cpp:365:28 [opt]
...
(lldb) p pcValues[0]
(U_8 *) 0x0000000000000003 ""
(lldb) p pcValues[1]
(U_8 *) 0x00000000013abfe7 "\xf5\xb8"
(lldb) p pcValues[2]
(U_8 *) 0x00000000013abfe7 "\xf5\xb8"
(lldb) p pcValues[3]
(U_8 *) 0x00000000013abfe7 "\xf5\xb8"
(lldb) p pcValues[4]
(U_8 *) 0x00000000013abfe7 "\xf5\xb8"
(lldb) p pcValues[5]
(U_8 *) 0x0000000000000006 ""
(lldb) p pcValues[6]
(U_8 *) 0x0000000000000006 ""
(lldb) p pcValues[7]
(U_8 *) 0x0000000000000006 ""
(lldb) up
(lldb) p walkState->pc
(U_8 *) 0x00000000013abfe7 "\xf5\xb8"
(lldb) p walkState->walkThread->pc
(U_8 *) 0x0000000000000006 ""

The assignment of walkState->pc = walkState->walkThread->pc appears between the points where pcValues[3] and pcValues[4] are captured, at which point its value is 0x13abfe7. By the time pcValues[5] is captured, walkState->walkThread->pc is 6.

@tajila, could someone on the VM team help investigate this further?

pshipton commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/38

pshipton commented 4 weeks ago

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_testList_0/30/

tajila commented 4 weeks ago

@babsingh Can you please take a look at this?

babsingh commented 4 weeks ago

I looked at the most recent core file from https://github.com/eclipse-openj9/openj9/issues/18910#issuecomment-2393932304.

There is only one virtual thread. It is being unmounted. As per #17303, there should be a retry (else if on line 83) to retrieve the vthread's stack, instead the else path is taken. Based upon the below DDR data, the conditions for taking the else if block are satisfied. Not sure, why the else block is being taken, especially when the J9VMThread is halted for inspection to avoid race conditions.


!continuationstack 0x00007fa5b05046a0 !j9vmcontinuation 0x00007fa5b05046a0 !j9object 0x000000000C921AA0 (Continuation) !j9object 0x000000000C9219A8 (VThread) -

> !j9object 0x000000000C9219A8
!J9Object 0x000000000C9219A8 {
    struct J9Class* clazz = !j9class 0x7FA5ACFDC800 // java/lang/VirtualThread
        ...
    Ljava/lang/Thread; carrierThread = !fj9object 0xb684200 (offset = 208) (java/lang/VirtualThread)

> !fj9object 0xb684200 | grep -i eetop
    J eetop = 0x00007FA5A9856F00 (offset = 8) (java/lang/Thread)

> !j9vmthread 0x00007FA5A9856F00 | grep -i currentContinuation
    0xa30: struct J9VMContinuation* currentContinuation = !j9vmcontinuation 0x0000000000000000

> !j9vmthread 0x00007FA5A9856F00 | grep -i threadobject
    0xa8: struct J9Object* threadObject = !j9object 0x000000000B684200 // jdk/internal/misc/CarrierThread
    0xa50: struct J9Object* carrierThreadObject = !j9object 0x000000000B684200 // jdk/internal/misc/CarrierThread

> !j9vmthread 0x00007FA5A9856F00 | grep -i publicflags
    0x98: UDATA publicFlags = 0x0000000000008281 (33409) // Thread is halted for inspection

# MainThread tries to get the vthread's stack

<7fa5aaa22900>  !j9method 0x00007FA5ACFBDD30   java/lang/Thread.getStackTraceImpl()Ljava/lang/Throwable;
<7fa5aaa22900>  !j9method 0x00007FA5ACFBDD10   java/lang/Thread.getStackTrace0()Ljava/lang/Object;
<7fa5aaa22900>  !j9method 0x00007FA5ACFBDCF0   java/lang/Thread.asyncGetStackTrace()[Ljava/lang/StackTraceElement;
<7fa5aaa22900>  !j9method 0x00007FA5ACFDC148   java/lang/VirtualThread.asyncGetStackTrace()[Ljava/lang/StackTraceElement;
<7fa5aaa22900>  !j9method 0x00007FA5ACFBDCD0   java/lang/Thread.getStackTrace()[Ljava/lang/StackTraceElement;
<7fa5aaa22900>  !j9method 0x00007FA5B0492BE8   GetStackTraceALotWhenPinned.main([Ljava/lang/String;)V

# Vthread is being unmounted

<7fa5a9856f00>  !j9method 0x00007FA5ACFDBDE8   java/lang/VirtualThread.unmount()V
<7fa5a9856f00>  !j9method 0x00007FA5ACFDBD08   java/lang/VirtualThread.runContinuation()V
<7fa5a9856f00>  !j9method 0x00007FA5B04E4C68   java/lang/VirtualThread$$Lambda/0x00000000b04acb48.run()V
<7fa5a9856f00>  !j9method 0x00007FA5B04E5C98   java/util/concurrent/ForkJoinTask$RunnableExecuteAction.compute()Ljava/lang/Void;
<7fa5a9856f00>  !j9method 0x00007FA5B04E5CF8   java/util/concurrent/ForkJoinTask$RunnableExecuteAction.compute()Ljava/lang/Object;
<7fa5a9856f00>  !j9method 0x00007FA5B04E5658   java/util/concurrent/ForkJoinTask$InterruptibleTask.exec()Z
<7fa5a9856f00>  !j9method 0x00007FA5B04CDB98   java/util/concurrent/ForkJoinTask.doExec()V
<7fa5a9856f00>  !j9method 0x00007FA5B04CFA40   java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(Ljava/util/concurrent/ForkJoinTask;I)V
<7fa5a9856f00>  !j9method 0x00007FA5B04A8468   java/util/concurrent/ForkJoinPool.runWorker(Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7fa5a9856f00>  !j9method 0x00007FA5B04A9E70   java/util/concurrent/ForkJoinWorkerThread.run()V

# VThread's carrier thread is yielded and hasn't resumed 

> !continuationstack 0x00007fa5b05046a0
<7fa5b05046a0>  !j9method 0x00007FA5ACFEF220   jdk/internal/vm/Continuation.yieldImpl(Z)Z
<7fa5b05046a0>  !j9method 0x00007FA5ACFEF0C0   jdk/internal/vm/Continuation.yield0()Z
<7fa5b05046a0>  !j9method 0x00007FA5ACFEF0A0   jdk/internal/vm/Continuation.yield(Ljdk/internal/vm/ContinuationScope;)Z
<7fa5b05046a0>  !j9method 0x00007FA5ACFDBE68   java/lang/VirtualThread.yieldContinuation()Z
<7fa5b05046a0>  !j9method 0x00007FA5ACFDC008   java/lang/VirtualThread.tryYield()V
<7fa5b05046a0>  !j9method 0x00007FA5ACFBD450   java/lang/Thread.yield()V
babsingh commented 4 weeks ago

I also looked at the core file which was inspected by @hzongaro in https://github.com/eclipse-openj9/openj9/issues/18910#issuecomment-2389640214. In this case, the virtual thread is being mounted. In both cases, the virtual thread is in an unsteady state (either being mounted or unmounted). This is indicated by inspectorCount = 0xFFFFFFFFFFFFFFFF (-1). @fengxue-IS Should we use if (-1 == inspectorCount) { retry; } in deciding whether to retry (ref: #17303)?


> !j9object 0x00000000E00E2AD8
!J9Object 0x00000000E00E2AD8 {
    struct J9Class* clazz = !j9class 0x49F6D00 // java/lang/VirtualThread
    ...
    Ljava/lang/Thread; carrierThread = !fj9object 0xe00f8a80 (offset = 136) (java/lang/VirtualThread)
    J inspectorCount = 0xFFFFFFFFFFFFFFFF (offset = 116) (java/lang/VirtualThread) <hidden>

> !j9object 0xe00f8a80
!J9Object 0x00000000E00F8A80 {
    struct J9Class* clazz = !j9class 0x4B3AE00 // jdk/internal/misc/CarrierThread
    ...
    J eetop = 0x0000000004C0AB00 (offset = 4) (java/lang/Thread)

> !j9vmthread 0x0000000004C0AB00 | grep -i currentcon
    0xa30: struct J9VMContinuation* currentContinuation = !j9vmcontinuation 0x00007F957C1089C0

> !j9vmthread 0x0000000004C0AB00 | grep -i threadobject
    0xa8: struct J9Object* threadObject = !j9object 0x00000000E00E2AD8 // java/lang/VirtualThread
    0xa50: struct J9Object* carrierThreadObject = !j9object 0x00000000E00F8A80 // jdk/internal/misc/CarrierThread

> !stack 0x0000000004C0AB00 // Virtual thread's stack
<4c0ab00>   !j9method 0x00000000049F6850   java/lang/VirtualThread.notifyJvmtiMount(Z)V
<4c0ab00>   !j9method 0x00000000049F63B0   java/lang/VirtualThread.yieldContinuation()Z
<4c0ab00>   !j9method 0x00000000049F6550   java/lang/VirtualThread.tryYield()V
<4c0ab00>   !j9method 0x00000000049DA710   java/lang/Thread.yield()V
<4c0ab00>   !j9method 0x0000000004BBC608   GetStackTraceALotWhenPinned.lambda$main$1(ILGetStackTraceALotWhenPinned$Barrier;)V
<4c0ab00>   !j9method 0x0000000004BBCC78   GetStackTraceALotWhenPinned$$Lambda/0x000000007eddb810.run()V
<4c0ab00>   !j9method 0x00000000049DAB30   java/lang/Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V
<4c0ab00>   !j9method 0x00000000049F62F0   java/lang/VirtualThread.run(Ljava/lang/Runnable;)V
<4c0ab00>   !j9method 0x0000000004BD7D80   java/lang/VirtualThread$VThreadContinuation$1.run()V
<4c0ab00>   !j9method 0x0000000004A07560   jdk/internal/vm/Continuation.enter(Ljdk/internal/vm/Continuation;)V
<4c0ab00>                           JNI call-in frame
<4c0ab00>                           Native method frame

> !continuationstack 0x00007f957c1089c0 // Carrier thread's stack
<7f957c1089c0>  !j9method 0x0000000004A07700   jdk/internal/vm/Continuation.enterImpl()Z
<7f957c1089c0>  !j9method 0x0000000004A07580   jdk/internal/vm/Continuation.run()V
<7f957c1089c0>  !j9method 0x00000000049F6250   java/lang/VirtualThread.runContinuation()V
<7f957c1089c0>  !j9method 0x0000000004BEE298   java/lang/VirtualThread$$Lambda/0x0000000000000000.run()V
<7f957c1089c0>  !j9method 0x0000000004BFC6A8   java/util/concurrent/ForkJoinTask$AdaptedRunnableAction.exec()Z
<7f957c1089c0>  !j9method 0x0000000004BEF018   java/util/concurrent/ForkJoinTask.doExec()I
<7f957c1089c0>  !j9method 0x0000000004BD7C48   java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(Ljava/util/concurrent/ForkJoinTask;Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7f957c1089c0>  !j9method 0x0000000004BD5410   java/util/concurrent/ForkJoinPool.scan(Ljava/util/concurrent/ForkJoinPool$WorkQueue;II)I
<7f957c1089c0>  !j9method 0x0000000004BD53F0   java/util/concurrent/ForkJoinPool.runWorker(Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7f957c1089c0>  !j9method 0x0000000004B3AB80   java/util/concurrent/ForkJoinWorkerThread.run()V
fengxue-IS commented 4 weeks ago

re https://github.com/eclipse-openj9/openj9/issues/18910#issuecomment-2394465606 and https://github.com/eclipse-openj9/openj9/issues/18910#issuecomment-2394503729:

There is a very unlikely code path where the continuation is already detached but the carrierThread have not been reset on the Vthread, by the time reaching check in #17303.

I didn't see any reason for the mount case to fail though as that work differently since the mount process is guarded more. regardless, adding if (-1 == inspectorCount) { retry; } at https://github.com/eclipse-openj9/openj9/blob/master/runtime/jcl/common/getstacktrace.c#L54-L66 should expand the guarded region and remove the issues. but this would come with some perf cost on the getStackTrace API

babsingh commented 4 weeks ago

Opened #20306. Running a personal build now. Will run a grinder afterwards to see if the failure is resolved by #20306.

github-actions[bot] commented 3 weeks ago

Issue Number: 18910 Status: Closed Actual Components: comp:vm, test failure, blocker, jdk22 Actual Assignees: No one :( PR Assignees: No one :(