Closed JasonFengJ9 closed 11 months ago
JDK21 aarch64_mac/ milestone 0(macaarch64rt8
)
[2023-09-22T18:52:07.426Z] variation: Mode650
[2023-09-22T18:52:07.426Z] JVM_OPTIONS: -XX:-UseCompressedOops
[2023-09-22T18:53:37.732Z] TEST: serviceability/jvmti/vthread/BreakpointInYieldTest/BreakpointInYieldTest.java
[2023-09-22T18:53:37.733Z] STDERR:
[2023-09-22T18:53:37.733Z] Unhandled exception
[2023-09-22T18:53:37.733Z] Type=Segmentation error vmState=0x0002000f
[2023-09-22T18:53:37.733Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2023-09-22T18:53:37.733Z] Handler1=0000000104B41094 Handler2=00000001049F51D8 InaccessibleAddress=FFFE8006AAA6A180
[2023-09-22T18:53:37.733Z] x0=FFFFA0014F9EF438 x1=000000016C2AD0A0 x2=FFFFA0014F9EF438 x3=0000000000000000
[2023-09-22T18:53:37.733Z] x4=000000016C2AD09C x5=000000016C2AD098 x6=000000016C2AD094 x7=0000000104BF7748
[2023-09-22T18:53:37.733Z] x8=000000016C2AD100 x9=00000001501177C8 x10=00000001501177E0 x11=0000000104C679CC
[2023-09-22T18:53:37.733Z] x12=0000000000000001 x13=0000000104C682A3 x14=0000000104C67BCC x15=0000000000000007
[2023-09-22T18:53:37.733Z] x16=000000016C2AD100 x17=FFFFA0029FB06C00 x18=0000000000000000 x19=00000000FFFFFFFF
[2023-09-22T18:53:37.733Z] x20=000000016C2AD95C x21=000000010491DEE0 x22=0000000000000000 x23=00000001501177B4
[2023-09-22T18:53:37.733Z] x24=FFFFA0014F9EF438 x25=0000000104BF76A4 x26=000000016C2AD0A0 x27=0000000000000000
[2023-09-22T18:53:37.733Z] x28=0000000000000003 x29(FP)=000000016C2AD900 x30(LR)=0000000104BF72C0 x31(SP)=000000016C2ACFD0
[2023-09-22T18:53:37.733Z] PC=0000000104BF74AC SP=000000016C2ACFD0
[2023-09-22T18:53:37.733Z] v0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v1 00000002801c8778 (f: 2149353216.000000, d: 5.305913e-314)
[2023-09-22T18:53:37.733Z] v2 000001be0000013e (f: 318.000000, d: 9.464101e-312)
[2023-09-22T18:53:37.733Z] v3 000001be000001be (f: 446.000000, d: 9.464101e-312)
[2023-09-22T18:53:37.733Z] v4 00000000000001be (f: 446.000000, d: 2.203533e-321)
[2023-09-22T18:53:37.733Z] v5 0000013e0000013e (f: 318.000000, d: 6.747947e-312)
[2023-09-22T18:53:37.733Z] v6 000000000000013e (f: 318.000000, d: 1.571129e-321)
[2023-09-22T18:53:37.733Z] v7 0000013e00000000 (f: 0.000000, d: 6.747947e-312)
[2023-09-22T18:53:37.733Z] v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2023-09-22T18:53:37.733Z] v17 3fd57028ca0c5555 (f: 3389805824.000000, d: 3.349707e-01)
[2023-09-22T18:53:37.733Z] v18 bf7aea0b0c8a2ba6 (f: 210381728.000000, d: -6.570857e-03)
[2023-09-22T18:53:37.733Z] v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2023-09-22T18:53:37.733Z] v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-09-22T18:53:37.733Z] Module=/Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_mac/openjdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2023-09-22T18:53:37.733Z] Module_base_address=0000000104B1C000 Symbol=mapLocalSet
[2023-09-22T18:53:37.733Z] Symbol_address=0000000104BF7420
[2023-09-22T18:53:37.733Z] Target=2_90_20230919_43 (Mac OS X 13.0)
[2023-09-22T18:53:37.733Z] CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
[2023-09-22T18:53:37.733Z] ----------- Stack Backtrace -----------
[2023-09-22T18:53:37.733Z] ---------------------------------------
[2023-09-22T18:53:37.733Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/22 14:53:11 - please wait.
[2023-09-22T18:53:56.143Z] serviceability_jvmti_j9_1_FAILED
@gacholio can you please take a look at this test_output
The tar file appears to be corrupt
j9build@736bb006f300:DOCKER-IMAGE $ tar xzf openjdk_test_output.tar.gz
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
j9build@736bb006f300:DOCKER-IMAGE $ gunzip openjdk_test_output.tar.gz
j9build@736bb006f300:DOCKER-IMAGE $ ls
openjdk_test_output.tar
j9build@736bb006f300:DOCKER-IMAGE $ tar xvf openjdk_test_output.tar
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
Looks like http download corrupts it - fetching with curl now.
A link to a closely matching JDK11 xa64 build would be helpful (for DDR). The SDK which produced the cores may also help, though I've never been able to run openj9 on my mac (millions of permission errors).
I think this is a better link https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Test_openjdk21_j9_extended.openjdk_aarch64_mac/4/openjdk_test_output.tar.gz
Link to aarch builds https://na-public.artifactory.swg-devops.com/ui/native/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/build-scripts/jobs/jdk21/jdk21-mac-aarch64-openj9/43/
Link to the equivalent xa64 https://na-public.artifactory.swg-devops.com/ui/native/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/build-scripts/jobs/jdk21/jdk21-linux-x64-openj9/43/
I think this was actually an x86 mac build.
I think this was actually an x86 mac build.
Are you talking about the original failure? or the one Jason posted 2 weeks ago?
I was looking at the start of the PR description. If the cores I have are from amac, the SDK won't help me anyway. I'll see if I can figure anything out from DDR. I also notice that no native stack traces appear in the javacore, which is unhelpful.
Here is the link for the core in the original failure: https://na-public.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/hyc-runtimes-jenkins.swg-devops.com/Test_openjdk21_j9_extended.openjdk_x86-64_mac_Personal/27/openjdk_test_output.tar.gz
Here is the link to the mac x86 build: https://na-public.artifactory.swg-devops.com/ui/native/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Build_JDK21_x86-64_mac_Personal/50/
I think the original failure was found in a personal build @JasonFengJ9 can confirm.
But using DDR on a newer JDK on xa64 should work
It's a crash in GC, so the native stacks probably aren't going to be very informative anyway.
I think the original failure was found in a personal build @JasonFengJ9 can confirm.
@tajila yes, the failure was from a personal build before JDK21 nightly/weekly builds run regularly. All codes were current with the main branches.
The mac x86 link above doesn't work (it's not even a link to a file!). Even signed in, it just presents me with thousands of unrelated artifacts.
You need to open it twice, it doesn't work the first time.
That worked, but what I really want is the linux x86-64 build. This looks like the right place to look:
but it hasn't been updated in years. Where should I be looking for the latest builds?
The latest nightly build is https://na-public.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/build-scripts/jobs/jdk11u/jdk11u-linux-x64-openj9/877/ibm-semeru-open-jdk_x64_linux_JDK11U_2023-10-10-22-01.tar.gz and you can truncate that to find older nightly builds. However they may not all be head stream builds.
Thanks, the mac x86 build is also working for me, now trying to get the original core downloaded (keeps failing half way through).
I have the original failure. Not surprisingly, the stack being walked appears to be in an unmounted continuation.
The DDR stack extensions don't appear to work on these stacks:
> !stackslots 0x00007000108A8CA0
Oct 11, 2023 2:33:25 P.M. com.ibm.j9ddr.vm29.events.DefaultEventListener corruptData
SEVERE: CDE thrown extracting initial stack walk state. walkThread = 0x00007000108A8CA0
com.ibm.j9ddr.NullPointerDereference: Memory Fault reading 0x00000000 :
at com.ibm.j9ddr.vm29.pointer.AbstractPointer.getLongAtOffset(AbstractPointer.java:456)
at com.ibm.j9ddr.vm29.pointer.generated.J9ThreadPointer.tid(Unknown Source)
at com.ibm.j9ddr.vm29.pointer.helper.J9ThreadHelper.getOSThread(J9ThreadHelper.java:60)
at com.ibm.j9ddr.vm29.j9.stackwalker.StackWalker$StackWalker_29_V0.walkStackFrames(StackWalker.java:171)
at com.ibm.j9ddr.vm29.j9.stackwalker.StackWalker.walkStackFrames(StackWalker.java:99)
Why DDR needs to know the OS thread ID is a mystery to me, but it's clear we don't fill that in for the stack-allocated threads used to drive the stack walker for unmounted continuations.
As the continuation is unmounted, the only way to find the failing stack is to examine every continuation using the new extension.
@babsingh Is there a command to list all of the continuations? I don't see anything obvious in the DDR help.
@babsingh Is there a command to list all of the continuations? I don't see anything obvious in the DDR help.
It will show in j9help
.
> !j9help | grep vthread
vthreads Lists virtual threads
> !j9help | grep contin
continuationstack <continuation> Walks the Java stack for <continuation>
continuationstackslots <continuation> Walks the Java stack (including objects) for <continuation>
Example:
!vthreads
Example output:
!continuationstack 0x00007fe78c0f9600 !j9vmcontinuation 0x00007fe78c0f9600 !j9object 0x0000000706401588 (Continuation) !j9object 0x0000000706400FB0 (VThread) - name1
!continuationstack 0x00007fe78c23aa80 !j9vmcontinuation 0x00007fe78c23aa80 !j9object 0x0000000706424F90 (Continuation) !j9object 0x0000000706424EF0 (VThread) - name2
!continuationstack 0x00007fe78c244ac0 !j9vmcontinuation 0x00007fe78c244ac0 !j9object 0x00000007064250D8 (Continuation) !j9object 0x0000000706425038 (VThread) - name3
!vthreads
provides no output at all in this core
> !vthreads
>
These cmds are only available in jdmpview for JDK21+. @fengxue-IS, fyi, if there is an actual bug.
openjdk version "21-internal" 2023-09-19
OpenJDK Runtime Environment (build 21-internal-adhoc.jenkins.BuildJDK21x86-64macPersonal)
Eclipse OpenJ9 VM (build master-7599bde8a13, JRE 21 Mac OS X amd64-64-Bit Compressed References 20230906_50 (JIT enabled, AOT enabled)
OpenJ9 - 7599bde8a13
OMR - 873ac5d377a
JCL - 154f45ddce4 based on jdk-21+35)
This is my JDK, and the vthreads command is clearly there, but does nothing.
Latest nightly build provides the same nothing.
This is my JDK, and the vthreads command is clearly there, but does nothing.
vthreads
iterates through the global linked list of continuation object lists
, which is maintained by the GC. If vthreads
doesn't show anything, then the list is probably empty and GC has collected the continuation objects. In https://github.com/eclipse-openj9/openj9/issues/18088#issue-1886046290, the native stack shows that the GC is scanning the objects and encounters a dead continuation. @gacholio Can you confirm if this statement is true; is the unmounted continuation mentioned in https://github.com/eclipse-openj9/openj9/issues/18088#issuecomment-1758308997 dead? DDR cmd: !isobjectalive <address>
. ++GC team @LinHu2016 @amicic @dmitripivkine for more insights.
class MM_GCExtensions : public MM_GCExtensionsBase {
private:
...
MM_ContinuationObjectList* continuationObjectLists; /**< The global linked list of continuation object lists. */
I say the continuation is unmounted because we're clearly in the fake thread/ELS case. I have no idea which continuation it is.
If the GC is scanning a dead continuation, then this should be looked at by the GC team.
The crash is in the local mapper. The method being mapped is:
> !bytecodes 0x13725790
Name: yield
Signature: (Ljdk/internal/vm/ContinuationScope;)Z
Access Flags (40009): public static
Internal Attribute Flags:
Max Stack: 2
Argument Count: 1
Temp Count: 2
0 breakpoint
1 dconst0
2 nop
3 invokeinterface2
5 invokeinterface 42 jdk/internal/access/JavaLangAccess.currentCarrierThread()Ljava/lang/Thread;
8 astore1
9 getstatic 14 jdk/internal/vm/Continuation.JLA Ljdk/internal/access/JavaLangAccess;
12 aload1
13 invokeinterface2
15 invokeinterface 43 jdk/internal/access/JavaLangAccess.getContinuation(Ljava/lang/Thread;)Ljdk/internal/vm/Continuation;
18 astore2
19 aload2
20 invokespecial 47 jdk/internal/vm/Continuation.yield0()Z
23 returnZ
If for some reason the mapper is being presented with the breakpointed method, that would explain the crash. This should not be the case: https://github.com/eclipse-openj9/openj9/blob/676b9a455892f883f7ecd78f53dfae50c5660646/runtime/vm/swalk.c#L873-L881
@LinHu2016 Please have a look at this to see why we're scanning a supposedly dead continuation. The core I'm looking at is:
@fengxue-IS The other interesting point is what ROM method is being passed to the local mapper. It should be the original (i.e. unbreakpointed), not the copy (breakpointed).
@gacholio There are two cores in this file failed in BreakpointInYieldTest. Which one I should look at? Would you please provide address of continuation object you suspect dead and should not be scanned?
for core.20230907.033504.84295.0001.dmp
:
there are three continuation objects in the list !mm_continuationobjectlist 0x00007FBCCA622E50
:
!j9object 0x35dcb4e0
!j9object 0x35dcb460
!j9object 0x35dbc478
all these objects are just forwarded, so alive
for core.20230907.033144.83304.0001.dmp
:
there are three continuation objects in the list !mm_continuationobjectlist 0x00007FD1D6C1D050
:
!j9object 0xffe2a768
!j9object 0xffe2a710
!j9object 0xffe095f0
all these objects are just forwarded, so alive
If we are talking about one of this objects I can check how it was discovered alive.
!vthreads
command shows nothing because in both cases there is only one non-empty sublist and it is under processing at the moment. I think DDR code can be improved to show not only result for _head
but also for temporary head _priorHead
.
"dead" continuation object == last unmounted continuation object, its vmRef (J9VMContinuation *)should be null and state & 0x2(J9_GC_CONTINUATION_STATE_FINISHED) should not be 0, in this case GC can not scan it (because J9VMContinuation == null), @dmitripivkine Could you please check vmRef and state fields for those continuation Objects? there is a case that GC happens during last unmount but before swap java stack, but in this special case the continuation should still be treated as mounted continuation, which we do not scan it during heap scan.
for first core:
> !j9object 0x35dcb4e0
!J9Object 0x0000000035DCB4E0 {
struct J9Class* clazz = !j9class 0x7FBCCE7B8F00 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FBCCB054D00 (offset = 8) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0x35d5d580 (offset = 32) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 40) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0x35d29988 (offset = 48) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0x35d5d6d8 (offset = 56) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 64) (jdk/internal/vm/Continuation)
J state = 0x0000000000000001 (offset = 16) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000001 (offset = 72) (jdk/internal/vm/Continuation)
J continuationLink = 0x0000000035D5D460 (offset = 24) (jdk/internal/vm/Continuation) <hidden>
}
> !j9object 0x0000000035dcb460
!J9Object 0x0000000035DCB460 {
struct J9Class* clazz = !j9class 0x7FBCCE7B8F00 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FBCCA659560 (offset = 8) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0x35d5d360 (offset = 32) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 40) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0x35d29988 (offset = 48) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0x35d5d4b8 (offset = 56) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 64) (jdk/internal/vm/Continuation)
J state = 0x0000000000000001 (offset = 16) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000001 (offset = 72) (jdk/internal/vm/Continuation)
J continuationLink = 0x0000000035D517C8 (offset = 24) (jdk/internal/vm/Continuation) <hidden>
}
> !j9object 0x0000000035dbc478
!J9Object 0x0000000035DBC478 {
struct J9Class* clazz = !j9class 0x7FBCCE7B8F00 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FBCCB054E40 (offset = 8) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0x35dbbc60 (offset = 32) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 40) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0x35db2b90 (offset = 48) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0x35dbd190 (offset = 56) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 64) (jdk/internal/vm/Continuation)
J state = 0x00007FBCCB849B01 (offset = 16) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000000 (offset = 72) (jdk/internal/vm/Continuation)
J continuationLink = 0x0000000000000000 (offset = 24) (jdk/internal/vm/Continuation) <hidden>
}
for second core:
> !j9object 0x00000000ffe2a768
!J9Object 0x00000000FFE2A768 {
struct J9Class* clazz = !j9class 0x138D9900 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FD1D6C50A10 (offset = 4) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0xfffe5148 (offset = 20) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 24) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0xfffc14d0 (offset = 28) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0xfffe5228 (offset = 32) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 36) (jdk/internal/vm/Continuation)
J state = 0x0000000000000001 (offset = 12) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000001 (offset = 40) (jdk/internal/vm/Continuation)
I continuationLink = 0xFFFE50A0 (offset = 44) (jdk/internal/vm/Continuation) <hidden>
}
> !j9object 0x00000000ffe2a710
!J9Object 0x00000000FFE2A710 {
struct J9Class* clazz = !j9class 0x138D9900 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FD1D5E1F260 (offset = 4) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0xfffe4ff8 (offset = 20) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 24) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0xfffc14d0 (offset = 28) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0xfffe50d8 (offset = 32) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 36) (jdk/internal/vm/Continuation)
J state = 0x0000000000000001 (offset = 12) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000001 (offset = 40) (jdk/internal/vm/Continuation)
I continuationLink = 0xFFFDCDE0 (offset = 44) (jdk/internal/vm/Continuation) <hidden>
}
> !j9object 0x00000000ffe095f0
!J9Object 0x00000000FFE095F0 {
struct J9Class* clazz = !j9class 0x138D9900 // java/lang/VirtualThread$VThreadContinuation
Object flags = 0x00000010;
I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
J vmRef = 0x00007FD1D5D35160 (offset = 4) (jdk/internal/vm/Continuation)
Ljava/lang/Thread; vthread = !fj9object 0xffe08fc8 (offset = 20) (jdk/internal/vm/Continuation)
[Ljava/lang/Object; scopedValueCache = !fj9object 0x0 (offset = 24) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0xffe0d8a8 (offset = 28) (jdk/internal/vm/Continuation)
Ljava/lang/Runnable; runnable = !fj9object 0xffe2b4e8 (offset = 32) (jdk/internal/vm/Continuation)
Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 36) (jdk/internal/vm/Continuation)
J state = 0x00000000138E6101 (offset = 12) (jdk/internal/vm/Continuation)
Z isAccessible = 0x00000000 (offset = 40) (jdk/internal/vm/Continuation)
I continuationLink = 0x00000000 (offset = 44) (jdk/internal/vm/Continuation) <hidden>
}
Looks like vmRef
is not NULL for any of them.
state
& 0x2 is 0
for both cores, two of continuations are unmounted, one is mounted, there are no concurrent states, pending mount state and finished state(all of continuations are started)
JDK21 x86-64_mac(macx64rt4
)
java version "21.0.1-beta" 2023-10-17
IBM Semeru Runtime Certified Edition 21.0.1+12-202310281508 (build 21.0.1-beta+12-202310281508)
Eclipse OpenJ9 VM 21.0.1+12-202310281508 (build master-7498dc04c, JRE 21 Mac OS X amd64-64-Bit Compressed References 20231028_24 (JIT enabled, AOT enabled)
OpenJ9 - 7498dc04c
OMR - 386a7080f
JCL - c06eaf638 based on jdk-21.0.1+12)
[2023-10-28T17:21:03.433Z] variation: Mode150
[2023-10-28T17:21:03.433Z] JVM_OPTIONS: -XX:+UseCompressedOops
[2023-10-28T17:23:07.015Z] TEST: serviceability/jvmti/vthread/BreakpointInYieldTest/BreakpointInYieldTest.java
[2023-10-28T17:23:07.016Z] STDERR:
[2023-10-28T17:23:07.016Z] Unhandled exception
[2023-10-28T17:23:07.016Z] Type=Segmentation error vmState=0x0002000f
[2023-10-28T17:23:07.016Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
[2023-10-28T17:23:07.016Z] Handler1=000000000D75EF30 Handler2=000000000D4AE970
[2023-10-28T17:23:07.016Z] RDI=0000000037D567D8 RSI=0000700002A372E0 RAX=0000000000000018 RBX=0000000000000018
[2023-10-28T17:23:07.016Z] RCX=FFFFA000352D6448 RDX=FFFFA0006D02CC20 R8=0000700002A372DC R9=0000700002A372D8
[2023-10-28T17:23:07.016Z] R10=0000000000000018 R11=0000000037D567F0 R12=0000700002A37270 R13=0000000000000007
[2023-10-28T17:23:07.016Z] R14=0000700002A37340 R15=0000000000000000
[2023-10-28T17:23:07.016Z] RIP=000000000D93EEA2 GS=0000 FS=0000 RSP=0000700002A371F8
[2023-10-28T17:23:07.016Z] RFlags=0000000000010296 CS=002B RBP=0000700002A37220 ERR=A002B00000000000
[2023-10-28T17:23:07.016Z] TRAPNO=000000000000000D CPU=B000000000000000 FAULTVADDR=00007FCCA002B000
[2023-10-28T17:23:07.016Z] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM1 00000000000000c6 (f: 198.000000, d: 9.782500e-322)
[2023-10-28T17:23:07.016Z] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM6 3fe4f87a3f5026e9 (f: 1062217472.000000, d: 6.553317e-01)
[2023-10-28T17:23:07.016Z] XMM7 402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
[2023-10-28T17:23:07.016Z] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-28T17:23:07.016Z] Module=/Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_mac/openjdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2023-10-28T17:23:07.016Z] Module_base_address=000000000D727000 Symbol=mapLocalSet
[2023-10-28T17:23:07.016Z] Symbol_address=000000000D93EE10
[2023-10-28T17:23:07.016Z] Target=2_90_20231028_24 (Mac OS X 13.2.1)
[2023-10-28T17:23:07.016Z] CPU=amd64 (12 logical CPUs) (0x400000000 RAM)
[2023-10-28T17:23:07.016Z] ----------- Stack Backtrace -----------
[2023-10-28T17:23:07.016Z] mapLocalSet+0x93 (0x000000000D93EEA3 [libj9vm29.dylib+0x217ea3])
[2023-10-28T17:23:07.016Z] j9localmap_LocalBitsForPC+0x5fb (0x000000000D93EC7B [libj9vm29.dylib+0x217c7b])
[2023-10-28T17:23:07.016Z] walkBytecodeFrameSlots+0x178 (0x000000000D7A1F58 [libj9vm29.dylib+0x7af58])
[2023-10-28T17:23:07.016Z] walkStackFrames+0x1136 (0x000000000D7A18B6 [libj9vm29.dylib+0x7a8b6])
[2023-10-28T17:23:07.016Z] walkContinuationStackFrames+0x1b1 (0x000000000D7B7971 [libj9vm29.dylib+0x90971])
[2023-10-28T17:23:07.016Z] _ZN28GC_VMThreadStackSlotIterator21scanContinuationSlotsEP10J9VMThreadP8J9ObjectPvPFvP8J9JavaVMPS3_S4_P16J9StackWalkStatePKvEbb+0x171 (0x000000000E0CEFA1 [libj9gc29.dylib+0xe5fa1])
[2023-10-28T17:23:07.016Z] _ZN20MM_ScavengerDelegate27scanContinuationNativeSlotsEP22MM_EnvironmentStandardP8J9Object21MM_ScavengeScanReasonb+0xc9 (0x000000000E0AB009 [libj9gc29.dylib+0xc2009])
[2023-10-28T17:23:07.016Z] _ZN20MM_ScavengerDelegate16getObjectScannerEP22MM_EnvironmentStandardP8J9ObjectPvm21MM_ScavengeScanReasonPb+0x2ed (0x000000000E0AB32D [libj9gc29.dylib+0xc232d])
[2023-10-28T17:23:07.016Z] _ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x5d6 (0x000000000E079096 [libj9gc29.dylib+0x90096])
[2023-10-28T17:23:07.016Z] _ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x1a6 (0x000000000E079AB6 [libj9gc29.dylib+0x90ab6])
[2023-10-28T17:23:07.016Z] _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x292 (0x000000000E079E82 [libj9gc29.dylib+0x90e82])
[2023-10-28T17:23:07.016Z] _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x77 (0x000000000E01E747 [libj9gc29.dylib+0x35747])
[2023-10-28T17:23:07.016Z] _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0xf6 (0x000000000E01E636 [libj9gc29.dylib+0x35636])
[2023-10-28T17:23:07.016Z] omrsig_protect+0x392 (0x000000000D4AD462 [libj9prt29.dylib+0x20462])
[2023-10-28T17:23:07.016Z] dispatcher_thread_proc+0x42 (0x000000000E01E6C2 [libj9gc29.dylib+0x356c2])
[2023-10-28T17:23:07.016Z] thread_wrapper+0x13a (0x000000000D41061A [libj9thr29.dylib+0xa61a])
[2023-10-28T17:23:07.016Z] _pthread_start+0x7d (0x00007FF815E1F259 [libsystem_pthread.dylib+0x6259])
[2023-10-28T17:23:07.016Z] ---------------------------------------
[2023-10-28T17:23:07.016Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/10/28 13:22:02 - please wait.
[2023-10-28T17:23:23.949Z] serviceability_jvmti_j9_0_FAILED
At this point we suspect jvmtiHelpers::fixBytecodesInAllStacks
is the cause as it doesn't fixup continuation stacks.
@JasonFengJ9 How reproduceable is this issue? I like to verify that updating the method above actually solves it.
Its also odd that we only see it on mac
Its also odd that we only see it on mac
This may have something to do with how freed memory is re-used.
@babsingh @fengxue-IS Given that we have exclusive, how do we go about finding all continuations?
Given that we have exclusive, how do we go about finding all continuations?
Through j9gc_flush_nonAllocationCaches_for_walk
and j9mm_iterate_all_continuation_objects
.
Thanks, I'll prototype a fix for this.
jitCodeBreakpointAdded
and jitCodeBreakpointRemoved
also need to be updated.
Failure link
From an internal build(
osxrt1
):Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
50x serviceability_jvmti_j9_0 internal grinder - reproduced 7/50
This seems similar to
FYI @babsingh