Open JasonFengJ9 opened 2 years ago
@JasonFengJ9 is this still failing
The test has been excluded, launched an internal grinder
LT Entering the loop that will replace dead threads with live ones until we run out of time.
LT stderr Exception in thread "Thread-7" java.lang.StackOverflowError: operating system stack overflow
LT stderr at java.base/java.lang.Thread.getAllThreads(Thread.java:2702)
LT stderr at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT stderr at java.base/java.lang.Thread.activeCount(Thread.java:2048)
LT stderr at java.base/java.lang.String.replace(String.java)
LT stderr at java.base/sun.invoke.util.BytecodeDescriptor.parseSig(BytecodeDescriptor.java:91)
Similar errors were reported on hotspot JDK19 as well HCRLateAttachWorkload_previewEnabled_0 and HCRLateAttachWorkload_previewEnabled_1 failing with StackOverflow Errors
The StackOverflowError
is still seen in OpenJ9 10x internal grinder
07:50:48 openjdk version "19-beta" 2022-09-20
07:50:48 IBM Semeru Runtime Open Edition 19+36-202210060535 (build 19-beta+36-202210060535)
07:50:48 Eclipse OpenJ9 VM 19+36-202210060535 (build master-c3faa0a56, JRE 19 Linux ppc64le-64-Bit Compressed References 20221006_50 (JIT enabled, AOT enabled)
07:50:48 OpenJ9 - c3faa0a56
07:50:48 OMR - 773e9b51a
07:50:48 JCL - b9e3daf582 based on jdk-19+36)
RI JDK19 passed 10x internal grinder
08:07:44 OpenJDK Runtime Environment Temurin-19+36 (build 19+36)
08:07:44 OpenJDK 64-Bit Server VM Temurin-19+36 (build 19+36, mixed mode, sharing)
FYI @tajila
is it all platforms?
Actually, this makes me think it may be a jit issue
LT stderr at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
LT stderr at java.base/java.lang.Thread.activeCount(Thread.java:2041)
LT stderr at java.base/java.lang.String.length(String.java) //<-- shouldnt be possible for length() to call activeCount()
LT stderr at java.base/java.lang.String.startsWith(String.java:3010)
@JasonFengJ9 Can you see if this reproduces with -Xint
Launched 10x -Xint internal grinder
Tried -Xint
as EXTRA_OPTIONS
, and also added mode 100
at https://github.com/JasonFengJ9/aqa-tests/blob/8b3a9959d7988a26673b7dce2135a9ebdb4b873b/system/otherLoadTest/playlist.xml#L294-L296
HCRLateAttachWorkload_previewEnabled_0_FAILED
w/ StackOverflowError
, from the javacore.20221018.131051.1747.0002.txt file, it wasn't running at -Xint
.
@llxia @Mesbah-Alam any insights on why the test won't take the grinder option or follow the mode specified?
From the console output, Mode100 did not get passed in:
14:24:18 variation: Mode100
14:24:18 JVM_OPTIONS: -Xdump:system:events=user
...
... -java-args='-Xdump:system:events=user'
This issue affects all system tests with Mode. It seems no Modes were passed in the system tests for the past month. The issue was introduced between Sep 18 (good build) - Sep 25 (bad build).
19:01:09 variation: Mode122
19:01:09 JVM_OPTIONS: -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs
...-java-args=' -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs -Xdump:system:events=user'
16:39:09 variation: Mode122
16:39:09 JVM_OPTIONS: -Xdump:system:events=user
...-java-args='-Xdump:system:events=user'
I suspect https://github.com/adoptium/aqa-tests/blob/d1c960685d0e4f043a11675608048eb7bd682f60/system/system.mk#L63 is incorrect. I think it is related to PR https://github.com/adoptium/aqa-tests/pull/3972
@JasonFengJ9 the above issue is resolved via https://github.com/adoptium/aqa-tests/pull/4052. Please try to rebase your branch and try again. Thanks
StackOverflowError: operating system stack overflow
also occurred at HCRLateAttachWorkload_previewEnabled_0_FAILED Mode100
which is -Xint
.
FYI @tajila
I added -Xdump:system:events=throw,filter=*StackOverflowError
and now the stackoverflow is gone, but it hangs instead.
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/28807/tapResults/
With -Xtrace:trigger=method{java/lang/StackOverflowError,sysdump}
08:16:20 AG stderr core file generated - /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16662681658036/HCRLateAttachWorkload_previewEnabled_0/20221020-051607-HCRLateAttachWorkload/results/core.20221020.051619.12033.0001.dmp
08:16:22 HCRLateAttachWorkload_previewEnabled_0_FAILED
Result file FYI @tajila
I see something very strange here:
The bytecodes for String::length() is:
public int length();
Code:
0: aload_0
1: getfield #50 // Field value:[B
4: arraylength
5: aload_0
6: invokevirtual #355 // Method coder:()B
9: ishr
10: ireturn
but when I look at the runtime version its
> !bytecodes 0x00000000004994D0
Name: length
Signature: ()I
Access Flags (50001): public
Internal Attribute Flags:
Max Stack: 12
Argument Count: 1
Temp Count: 0
0 invokestatic 31 java/lang/Thread.activeCount()I
3 pop
4 aload0getfield
5 getfield 27 java/lang/String.value [B
8 arraylength
9 aload0
10 invokevirtual 139 java/lang/String.coder()B
13 ishr
14 return1
So there are some extra bytecodes at the start
The class is retransformed, the original bytecodes before the transformation is correct
> !bytecodes 0x00000000000A4F50
Name: length
Signature: ()I
Access Flags (50001): public
Internal Attribute Flags:
Max Stack: 2
Argument Count: 1
Temp Count: 0
0 aload0getfield
1 getfield 18 java/lang/String.value [B
4 arraylength
5 aload0
6 invokevirtual 117 java/lang/String.coder()B
9 ishr
10 return1
I dont know much about this test, but if that transformation is intended then the StackOverflow makes sense.
Based on the README, https://github.com/adoptium/aqa-systemtest/blob/0eb9f57f198fe194c070295c261e8bafeaed08aa/openjdk.test.debugging/docs/HCR_README.md
The test randomly adds a method to String API, I think in this particular case there is nothing we can do about it because it is infinitely recursive. I think we can close and re-enable the test, with a different seed value we shouldn't see the failure.
@JasonFengJ9 can you please unexclude the test
HCRLateAttachWorkload_previewEnabled
has been un-excluded.
The test is failing on all platforms, https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK19/194/ It needs to be excluded again, today if possible.
Example failure, https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_aarch64_linux_Nightly/43
LT stderr *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 873
LT stderr Exception in thread "Thread-11" java/lang/StackOverflowError: operating system stack overflow
LT stderr at java/lang/Thread.getAllThreads (java.base@19.0.1-internal/Thread.java:2704)
LT stderr at java/lang/ThreadGroup.activeCount (java.base@19.0.1-internal/ThreadGroup.java:377)
LT stderr at java/lang/Thread.activeCount (java.base@19.0.1-internal/Thread.java:2046)
LT stderr at java/lang/String.startsWith (java.base@19.0.1-internal/String.java:4294967295)
LT stderr at java/lang/invoke/MethodType.fromDescriptor (java.base@19.0.1-internal/MethodType.java:1212)
LT stderr at java/lang/invoke/MethodType.fromMethodDescriptorString (java.base@19.0.1-internal/MethodType.java:1195)
LT stderr at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString (java.base@19.0.1-internal/MethodTypeHelper.java:256)
LT stderr at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic (java.base@19.0.1-internal/MethodHandleResolver.java:168)
LT stderr at java/lang/Thread.getAllThreads (java.base@19.0.1-internal/Thread.java:2704)
LT stderr at java/lang/ThreadGroup.activeCount (java.base@19.0.1-internal/ThreadGroup.java:377)
LT stderr at java/lang/Thread.activeCount (java.base@19.0.1-internal/Thread.java:2046)
LT stderr at java/lang/String.startsWith (java.base@19.0.1-internal/String.java:4294967295)
LT stderr at java/lang/invoke/MethodType.fromDescriptor (java.base@19.0.1-internal/MethodType.java:1212)
LT stderr at java/lang/invoke/MethodType.fromMethodDescriptorString (java.base@19.0.1-internal/MethodType.java:1195)
LT stderr at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString (java.base@19.0.1-internal/MethodTypeHelper.java:256)
LT stderr at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic (java.base@19.0.1-internal/MethodHandleResolver.java:168)
LT stderr at java/lang/Thread.getAllThreads (java.base@19.0.1-internal/Thread.java:2704)
Segmentation error vmState=0x00000000
at an internal build(win16x64rt2-2
)
[2022-11-05T21:25:02.244Z] variation: Mode150
[2022-11-05T21:25:02.681Z] JVM_OPTIONS: -XX:+UseCompressedOops
[2022-11-05T21:25:17.971Z] LT 14:25:17.847 - Starting thread. Suite=0 thread=2
[2022-11-05T21:25:19.565Z] LT stderr Unhandled exception
[2022-11-05T21:25:19.566Z] LT stderr Type=Segmentation error vmState=0x00000000
[2022-11-05T21:25:19.566Z] LT stderr Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFBAC436F38 ContextFlags=0010005f
[2022-11-05T21:25:19.566Z] LT stderr Handler1=00007FFBAC4EC230 Handler2=00007FFBB23EAA50 =0000000000000000
[2022-11-05T21:25:19.566Z] LT stderr RDI=0000002DC8F5F168 RSI=00000007FFD38D38 RAX=0000000000435B00 RBX=0000014DB24D3FC0
[2022-11-05T21:25:19.566Z] LT stderr RCX=0000000040000000 RDX=000000000065BF58 R8=0000002DC8F5F178 R9=0000002DC8F5F190
[2022-11-05T21:25:19.566Z] LT stderr R10=00007FFBAC5DB374 R11=0000002DC8F5F180 R12=000000000065BF50 R13=0000000000000004
[2022-11-05T21:25:19.566Z] LT stderr R14=0000002DC8F5F170 R15=00000007FFD15198
[2022-11-05T21:25:19.566Z] LT stderr RIP=00007FFBAC436F38 RSP=0000002DC8F5EE60 RBP=FFFA71ABFFFA719A EFLAGS=0000000000010217
[2022-11-05T21:25:19.566Z] LT stderr FS=0053 ES=002B DS=002B
[2022-11-05T21:25:19.566Z] LT stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM1 00000000003b9560 (f: 3904864.000000, d: 1.929259e-317)
[2022-11-05T21:25:19.566Z] LT stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-05T21:25:19.566Z] LT stderr Module=C:\Users\jenkins\workspace\Test_openjdk19_j9_extended.system_x86-64_windows_testList_0\openjdkbinary\j2sdk-image\bin\default\j9vm29.dll
[2022-11-05T21:25:19.566Z] LT stderr Module_base_address=00007FFBAC420000 Offset_in_DLL=0000000000016f38
[2022-11-05T21:25:19.566Z] LT stderr Target=2_90_20221018_69 (Windows Server 2016 10.0 build 14393)
[2022-11-05T21:25:19.566Z] LT stderr CPU=amd64 (8 logical CPUs) (0x3fff78000 RAM)
[2022-11-05T21:25:19.566Z] LT stderr ----------- Stack Backtrace -----------
[2022-11-05T21:25:19.566Z] STF 14:25:19.245 - Found dump at: C:\Users\jenkins\workspace\Test_openjdk19_j9_extended.system_x86-64_windows_testList_0\aqa-tests\TKG\output_16676811979167\HCRLateAttachWorkload_previewEnabled_0\20221105-142503-HCRLateAttachWorkload\results\core.20221105.142519.4344.0001.dmp
[2022-11-05T21:25:19.566Z] LT stderr (0x00007FFBAC436F38 [j9vm29+0x16f38])
[2022-11-05T21:25:19.566Z] LT stderr (0xFFFFFFFFFFFFFFFD)
[2022-11-05T21:25:19.566Z] LT stderr (0x0000002DC8F5F180)
[2022-11-05T21:25:19.566Z] LT stderr (0x0000002DC8F5F170)
[2022-11-05T21:25:19.566Z] LT stderr (0x0000014DBDE59A68)
[2022-11-05T21:25:19.566Z] LT stderr (0x0000014DB2528BB0)
[2022-11-05T21:25:19.566Z] LT stderr (0x0000014DB2513EC0)
[2022-11-05T21:25:19.566Z] LT stderr ---------------------------------------
[2022-11-05T21:25:19.566Z] LT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/05 14:25:19 - please wait.
[2022-11-05T21:25:51.048Z] HCRLateAttachWorkload_previewEnabled_0_FAILED
Failure link
From an internal build
job/Test_openjdknext_j9_extended.system_aarch64_linux_Personal/1/tapResults/
(cent7-aarch64-6
):Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)