search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

JDK19 java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0 - Segmentation error vmState=0x0002000f #16351

Closed JasonFengJ9 closed 1 year ago

JasonFengJ9 commented 1 year ago

Failure link

From an internal build(rhel7lert-11):

openjdk version "19.0.1" 2022-10-18
IBM Semeru Runtime Open Edition 19.0.1+10 (build 19.0.1+10)
Eclipse OpenJ9 VM 19.0.1+10 (build master-5e4baa709, JRE 19 Linux ppc64le-64-Bit Compressed References 20221018_78 (JIT enabled, AOT enabled)
OpenJ9   - 5e4baa709
OMR      - fe4c3b9b5
JCL      - 720d535776 based on jdk-19.0.1+10)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-11-19T17:54:36.618Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode150
[2022-11-19T17:54:36.618Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops 

[2022-11-19T18:37:57.679Z] TEST: java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0

[2022-11-19T18:37:57.680Z] STDERR:
[2022-11-19T18:37:57.680Z] JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/core.20221119.103746.5578.0001.dmp' through com.ibm.jvm.Dump.systemDumpToFile
[2022-11-19T18:37:57.680Z] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/core.20221119.103746.5578.0001.dmp
[2022-11-19T18:37:57.680Z] JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/javacore.20221119.103748.5578.0002.txt' through com.ibm.jvm.Dump.javaDumpToFile
[2022-11-19T18:37:57.680Z] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work/scratch/1/javacore.20221119.103748.5578.0002.txt
[2022-11-19T18:37:57.680Z] Unhandled exception
[2022-11-19T18:37:57.680Z] Type=Segmentation error vmState=0x0002000f
[2022-11-19T18:37:57.680Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2022-11-19T18:37:57.680Z] Handler1=00003FFFA53B0140 Handler2=00003FFFA52D8AA0
[2022-11-19T18:37:57.680Z] R0=0000000000000008 R1=00003FFF7C36BA20 R2=00003FFFA5C07400 R3=00000000E01B3178
[2022-11-19T18:37:57.680Z] R4=0000000100000000 R5=0000000000520042 R6=0000000000000010 R7=0000000000000020
[2022-11-19T18:37:57.680Z] R8=0000000000000030 R9=00003FFF7C36BCB0 R10=0000000000000008 R11=00000000E01FE700
[2022-11-19T18:37:57.680Z] R12=000000000000A400 R13=00003FFF7C376900 R14=00000000FFFB4A78 R15=00000000FFFB4A78
[2022-11-19T18:37:57.680Z] R16=0000000000000000 R17=0000000000000001 R18=00003FFF7C36BCE0 R19=0000000000000000
[2022-11-19T18:37:57.680Z] R20=00003FFFA0090270 R21=00003FFF7C36BCB0 R22=00000006000B7700 R23=00000000E06D31C8
[2022-11-19T18:37:57.680Z] R24=0000000000000009 R25=0000000000000000 R26=00000000E01B3178 R27=00000000E01B3178
[2022-11-19T18:37:57.680Z] R28=00003FFF7C36BCB0 R29=00003FFF985410D0 R30=00003FFFA008FFC0 R31=00003FFF28001AC8
[2022-11-19T18:37:57.680Z] NIP=00003FFFA5ACABC0 MSR=800000010280F033 ORIG_GPR3=C0000000000027E0 CTR=0000000000009A95
[2022-11-19T18:37:57.680Z] LINK=00003FFFA4DA8C40 XER=0000000000000000 CCR=0000000044844248 SOFTE=0000000000000001
[2022-11-19T18:37:57.680Z] TRAP=0000000000000300 DAR=0000000100000000 dsisr=0000000040000000 RESULT=0000000000000000
[2022-11-19T18:37:57.680Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR1 000000000052057e (f: 5375358.000000, d: 2.655780e-317)
[2022-11-19T18:37:57.680Z] FPR2 3fd6e80000000000 (f: 0.000000, d: 3.579102e-01)
[2022-11-19T18:37:57.680Z] FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR4 3fe3333340000000 (f: 1073741824.000000, d: 6.000000e-01)
[2022-11-19T18:37:57.680Z] FPR5 3fd6e850e0000000 (f: 3758096384.000000, d: 3.579294e-01)
[2022-11-19T18:37:57.680Z] FPR6 3eb742873eb74287 (f: 1052197504.000000, d: 1.386397e-06)
[2022-11-19T18:37:57.680Z] FPR7 40365ba140000000 (f: 1073741824.000000, d: 2.235793e+01)
[2022-11-19T18:37:57.680Z] FPR8 3e66376980000000 (f: 2147483648.000000, d: 4.138137e-08)
[2022-11-19T18:37:57.680Z] FPR9 4000000000000000 (f: 0.000000, d: 2.000000e+00)
[2022-11-19T18:37:57.680Z] FPR10 3fae368d40000000 (f: 1073741824.000000, d: 5.900995e-02)
[2022-11-19T18:37:57.680Z] FPR11 bfa1254480000000 (f: 2147483648.000000, d: -3.348745e-02)
[2022-11-19T18:37:57.680Z] FPR12 4aa40afc4aa40afc (f: 1252264704.000000, d: 3.749472e+51)
[2022-11-19T18:37:57.680Z] FPR13 3fdb9b2820000000 (f: 536870912.000000, d: 4.313450e-01)
[2022-11-19T18:37:57.680Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-19T18:37:57.680Z] Module=/lib64/libc.so.6
[2022-11-19T18:37:57.680Z] Module_base_address=00003FFFA5A20000
[2022-11-19T18:37:57.680Z] Target=2_90_20221018_78 (Linux 3.10.0-1160.80.1.el7.ppc64le)
[2022-11-19T18:37:57.680Z] CPU=ppc64le (4 logical CPUs) (0x1de3d0000 RAM)
[2022-11-19T18:37:57.680Z] ----------- Stack Backtrace -----------
[2022-11-19T18:37:57.680Z] protectedBacktrace+0x24 (0x00003FFFA52D2EB4 [libj9prt29.so+0x32eb4])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] omrintrospect_backtrace_thread_raw+0xe0 (0x00003FFFA52D34D0 [libj9prt29.so+0x334d0])
[2022-11-19T18:37:57.680Z] protectedIntrospectBacktraceThread+0x24 (0x00003FFFA52D2AF4 [libj9prt29.so+0x32af4])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] omrintrospect_backtrace_thread+0x94 (0x00003FFFA52D2C04 [libj9prt29.so+0x32c04])
[2022-11-19T18:37:57.680Z] generateDiagnosticFiles+0xd4 (0x00003FFFA53AFB14 [libj9vm29.so+0x3fb14])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] structuredSignalHandler+0x194 (0x00003FFFA53AFE74 [libj9vm29.so+0x3fe74])
[2022-11-19T18:37:57.680Z] structuredSignalHandlerVM+0x58 (0x00003FFFA53B0198 [libj9vm29.so+0x40198])
[2022-11-19T18:37:57.680Z] mainSynchSignalHandler+0x294 (0x00003FFFA52D8D34 [libj9prt29.so+0x38d34])
[2022-11-19T18:37:57.680Z] __kernel_sigtramp_rt64+0x0 (0x00003FFFA5D10478)
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0x360 (0x00003FFFA4DA8720 [libj9gc29.so+0x1e8720])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x978 (0x00003FFFA4DA4F18 [libj9gc29.so+0x1e4f18])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x114 (0x00003FFFA4DA55C4 [libj9gc29.so+0x1e55c4])
[2022-11-19T18:37:57.680Z] _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x2b4 (0x00003FFFA4DA5AE4 [libj9gc29.so+0x1e5ae4])
[2022-11-19T18:37:57.680Z] _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x1c (0x00003FFFA4E1DD3C [libj9gc29.so+0x25dd3c])
[2022-11-19T18:37:57.680Z] _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x2b8 (0x00003FFFA4D40328 [libj9gc29.so+0x180328])
[2022-11-19T18:37:57.680Z] _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x160 (0x00003FFFA4D3F6F0 [libj9gc29.so+0x17f6f0])
[2022-11-19T18:37:57.680Z] omrsig_protect+0x3f4 (0x00003FFFA52D9F74 [libj9prt29.so+0x39f74])
[2022-11-19T18:37:57.680Z] dispatcher_thread_proc+0x50 (0x00003FFFA4D3EDB0 [libj9gc29.so+0x17edb0])
[2022-11-19T18:37:57.680Z] thread_wrapper+0x190 (0x00003FFFA526CBC0 [libj9thr29.so+0xcbc0])
[2022-11-19T18:37:57.680Z] start_thread+0x104 (0x00003FFFA5C48CD4 [libpthread.so.0+0x8cd4])
[2022-11-19T18:37:57.680Z] clone+0xe4 (0x00003FFFA5B47F14 [libc.so.6+0x127f14])
[2022-11-19T18:37:57.680Z] ---------------------------------------
[2022-11-19T18:37:57.680Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/19 10:37:49 - please wait.

[2022-11-19T18:37:57.681Z] TEST RESULT: Error. Program `/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 965937ms).
[2022-11-19T18:37:57.681Z] --------------------------------------------------
[2022-11-19T18:43:14.332Z] Test results: passed: 845; error: 1
[2022-11-19T18:43:39.475Z] Report written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/report/html/report.html
[2022-11-19T18:43:39.475Z] Results written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_testList_1/aqa-tests/TKG/output_16688804752890/jdk_lang_0/work
[2022-11-19T18:43:39.475Z] Error: Some tests failed or other problems occurred.
[2022-11-19T18:43:39.475Z] 
[2022-11-19T18:43:39.475Z] jdk_lang_0_FAILED

50x internal grinder: ubu18le-rt1-6 failure, rhel7lert-1-10 failure

FYI @dmitripivkine

dmitripivkine commented 1 year ago

This is another manifestation of crashes in Scavenger we collecting in https://github.com/eclipse-openj9/openj9/issues/16249

> !j9javavm 0x00003FFFA001CEC0 | grep liveVirtualThreadList
    0x4f88: struct J9Object** liveVirtualThreadList = !j9x 0x00003FFFA00A1888
> !j9x 0x00003FFFA00A1888
0x3FFFA00A1888 :  00000000ffd8dbf0 00000000e0109860 [ ........`....... ]

> !j9object 0x00000000ffd8dbf0
!J9Object 0x00000000FFD8DBF0 {
    struct J9Class* clazz = !j9class 0x101C00 // java/lang/VirtualThread
    Object flags = 0x0000002A;
    I lockword = 0x00000008 (offset = 0) (java/lang/Object) <hidden>
    J eetop = 0x0000000000000000 (offset = 4) (java/lang/Thread)
    J tid = 0x0000000000000000 (offset = 12) (java/lang/Thread)
    Ljava/lang/String; name = !fj9object 0x0 (offset = 36) (java/lang/Thread)
    Z interrupted = 0x00000000 (offset = 92) (java/lang/Thread)
    Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x0 (offset = 40) (java/lang/Thread)
    Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x0 (offset = 44) (java/lang/Thread)
    Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 48) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 52) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 56) (java/lang/Thread)
    Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 60) (java/lang/Thread)
    Ljava/lang/Object; interruptLock = !fj9object 0x0 (offset = 64) (java/lang/Thread)
    Ljava/lang/Object; parkBlocker = !fj9object 0x0 (offset = 68) (java/lang/Thread)
    Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 72) (java/lang/Thread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 76) (java/lang/Thread)
    Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 80) (java/lang/Thread)
    J threadLocalRandomSeed = 0x0000000000000000 (offset = 20) (java/lang/Thread)
    I threadLocalRandomProbe = 0x00000000 (offset = 96) (java/lang/Thread)
    I threadLocalRandomSecondarySeed = 0x00000000 (offset = 100) (java/lang/Thread)
    Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x0 (offset = 84) (java/lang/Thread)
    Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 88) (java/lang/Thread)
    Z started = 0x00000000 (offset = 104) (java/lang/Thread)
    Z stopCalled = 0x00000000 (offset = 108) (java/lang/Thread)
    J tls = 0x0000000000000000 (offset = 28) (java/lang/Thread) <hidden>
    Ljava/util/concurrent/Executor; scheduler = !fj9object 0x0 (offset = 124) (java/lang/VirtualThread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 128) (java/lang/VirtualThread)
    Ljava/lang/Runnable; runContinuation = !fj9object 0x0 (offset = 132) (java/lang/VirtualThread)
    I state = 0x00000000 (offset = 112) (java/lang/VirtualThread)
    Z parkPermit = 0x00000000 (offset = 152) (java/lang/VirtualThread)
    Ljava/lang/Thread; carrierThread = !fj9object 0x0 (offset = 136) (java/lang/VirtualThread)
    Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x0 (offset = 140) (java/lang/VirtualThread)
    Ljava/lang/VirtualThread; linkNext = !fj9object 0xfffb4a78 (offset = 148) (java/lang/VirtualThread) <hidden>
    Ljava/lang/VirtualThread; linkPrevious = !fj9object 0xfffb4a78 (offset = 144) (java/lang/VirtualThread) <hidden>
    J inspectorCount = 0x0000000000000000 (offset = 116) (java/lang/VirtualThread) <hidden>
    I isSuspendedByJVMTI = 0x00000000 (offset = 156) (java/lang/VirtualThread) <hidden>
}
> !fj9object 0xfffb4a78
Unable to read object clazz at 0x00000000FFFB4A78 (clazz = 0x00000000E01B3100)
> !j9x 0xfffb4a00,100,4
0xFFFB4A00 :  fffb1830 fffb1868 fffb18a8 fffb18d8 [ 0...h........... ]
0xFFFB4A10 :  fffb1910 fffb1950 fffb1990 fffb19c0 [ ....P........... ]
0xFFFB4A20 :  fffb1a00 fffb1a38 fffb1c30 fffb1c70 [ ....8...0...p... ]
0xFFFB4A30 :  fffb1ca8 fffb1cd8 fffb1d10 fffb1d50 [ ............P... ]
0xFFFB4A40 :  fffb1d80 fffb1dc0 fffb1df8 fffb1e28 [ ............(... ]
0xFFFB4A50 :  fffb1e60 fffb1ea0 fffb1ed8 fffb1f08 [ `............... ]
0xFFFB4A60 :  fffb1f40 fffb1f78 fffb1fb8 fffb1fe8 [ @...x........... ]
0xFFFB4A70 :  fffb2020 fffb2060 e01b317c 00000000 [   ..` ..|1...... ]
0xFFFB4A80 :  fffb2108 fffb2138 fffb2170 fffb21a8 [ .!..8!..p!...!.. ]
0xFFFB4A90 :  fffb21e8 fffb2220 fffb2260 fffb2290 [ .!.. "..`"...".. ]
0xFFFB4AA0 :  fffb22c8 fffb2300 fffb2348 fffb2380 [ ."...#..H#...#.. ]
0xFFFB4AB0 :  fffb23b8 fffb23f8 fffb26c8 fffb2700 [ .#...#...&...'.. ]
0xFFFB4AC0 :  fffb2740 fffb2778 fffb27a8 fffb27e8 [ @'..x'...'...'.. ]
0xFFFB4AD0 :  fffb2820 fffb2850 fffb2890 fffb28c8 [  (..P(...(...(.. ]
0xFFFB4AE0 :  fffb28f8 fffb2938 fffb2970 fffb29a0 [ .(..8)..p)...).. ]
0xFFFB4AF0 :  fffb29e0 fffb2a18 fffb2a48 fffb2a88 [ .)...*..H*...*.. ]
> !j9object 0xe01b3178
Unable to read object clazz at 0x00000000E01B3178 (clazz = 0x00000000E01B3100)
> !j9x 0x00000000E01B3100,100,4
0xE01B3100 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3110 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3120 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3130 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3140 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3150 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3160 :  00000000 00000000 00000000 00000000 [ ................ ]
0xE01B3170 :  00000000 00000000 e01b317c 00000000 [ ........|1...... ]
0xE01B3180 :  fffb2108 fffb2138 fffb2170 fffb21a8 [ .!..8!..p!...!.. ]
0xE01B3190 :  fffb21e8 fffb2220 fffb2260 fffb2290 [ .!.. "..`"...".. ]
0xE01B31A0 :  fffb22c8 fffb2300 fffb2348 fffb2380 [ ."...#..H#...#.. ]
0xE01B31B0 :  fffb23b8 fffb23f8 fffb26c8 fffb2700 [ .#...#...&...'.. ]
0xE01B31C0 :  fffb2740 fffb2778 fffb27a8 fffb27e8 [ @'..x'...'...'.. ]
0xE01B31D0 :  fffb2820 fffb2850 fffb2890 fffb28c8 [  (..P(...(...(.. ]
0xE01B31E0 :  fffb28f8 fffb2938 fffb2970 fffb29a0 [ .(..8)..p)...).. ]
0xE01B31F0 :  fffb29e0 fffb2a18 fffb2a48 fffb2a88 [ .)...*..H*...*.. ]
>

We are trying to copy bogus object !fj9object 0xfffb4a78 and crashed

Closing as duplicate of https://github.com/eclipse-openj9/openj9/issues/16249

tajila commented 1 year ago

10x grinder with trace logs https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29690/

tajila commented 1 year ago

Ill reopen this as the other issue contains the jit invalid address details. We can keep discussion on corrupt vthread list here

tajila commented 1 year ago

@babsingh can you please take a look

JasonFengJ9 commented 1 year ago

The grinder reproduced the Segmentation error vmState=0x0002000f twice: ubu18le-rt1-6 failure, rhel7lert-1-10 failure

Grinder 4(rhel8le-rt1-10) had an assertion j9mm.141 * ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1622: ((false))

09:03:45  STDERR:
09:03:45  JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/core.20221122.060328.1566414.0001.dmp' through com.ibm.jvm.Dump.systemDumpToFile
09:03:45  JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/core.20221122.060328.1566414.0001.dmp
09:03:45  JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/javacore.20221122.060329.1566414.0002.txt' through com.ibm.jvm.Dump.javaDumpToFile
09:03:45  JVMDUMP010I Java dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1669124827859/jdk_custom_0/work/scratch/0/javacore.20221122.060329.1566414.0002.txt
09:03:45  Corruption in Evacuate at 00000000FFFBC1C8: calculated object size 3996785944479924229 larger than available 278072, Forwarded Header at 00007FFF8C3FBC50
09:03:45  14:03:31.006 0x23ca00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1622: ((false))
09:03:45  JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/11/22 06:03:31 - please wait.
tajila commented 1 year ago

@JasonFengJ9 can you reproduce with -Xtrace:maximal={j9scar{level5}},buffers=2m, we added some new tracepoints to help diagnose this issue

JasonFengJ9 commented 1 year ago

@tajila 50x grinder -Xtrace:maximal={j9scar{level5}},buffers=2m just was launched.

dmitripivkine commented 1 year ago

Another failure in the grinder (launched for different defect) https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29676/

> !gpinfo
Failing Thread: !j9vmthread 0x3fef4002400
Failing Thread ID: 0x1db01 (121601)
gpInfo:
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF94B44858 Handler2=000003FF94A31CF0 InaccessibleAddress=0408040804080000
gpr0=000003FF9027C210 gpr1=000003FF9003E158 gpr2=000003FF9005F728 gpr3=000003FEF4004338
gpr4=000003FF944FE6C0 gpr5=000003FF944FE478 gpr6=000003FF944FE480 gpr7=000003FF900AE170
gpr8=000003FF944FE6D8 gpr9=000003FEF4004338 gpr10=000003FF900AE170 gpr11=0408040804080400
gpr12=000003FF95726F88 gpr13=000003FF944FE6C0 gpr14=000003FF9432C368 gpr15=000003FF944FE2E8
psw=000003FF9435625C mask=0705100180000000 fpc=00080000 bea=000003FF9432C362
fpr0 429841d9d1da893c (f: 3520760064.000000, d: 6.667777e+12)
fpr1 000003ff944fe088 (f: 2488262912.000000, d: 2.172031e-311)
fpr2 3e3ab2831be3a8dd (f: 467904736.000000, d: 6.215953e-09)
fpr3 0000000000110000 (f: 1114112.000000, d: 5.504445e-318)
fpr4 402bb9d3beb8c86b (f: 3199780864.000000, d: 1.386294e+01)
fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr6 3f33240000000000 (f: 0.000000, d: 2.920628e-04)
fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr8 0000000000041000 (f: 266240.000000, d: 1.315400e-318)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 000003ff944bf000 (f: 2488004608.000000, d: 2.172031e-311)
fpr11 000003ffd3ffb4e0 (f: 3556750592.000000, d: 2.172559e-311)
fpr12 0005ee048c602b82 (f: 2355112960.000000, d: 8.246342e-309)
fpr13 000003fee8045d58 (f: 3892600064.000000, d: 2.170603e-311)
fpr14 000003ff943c65e0 (f: 2486986240.000000, d: 2.172030e-311)
fpr15 000003fee809ab18 (f: 3892947712.000000, d: 2.170603e-311)
Module=/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
Module_base_address=000003FF94180000

> !findvm
!j9javavm 0x000003FF90038DA0
> !j9javavm 0x000003FF90038DA0 | grep liveVirtualThreadList
    0x4f40: struct J9Object** liveVirtualThreadList = !j9x 0x000003FF900C0388
    0x4f48: struct J9ThreadMonitor* liveVirtualThreadListMutex = !j9threadmonitor 0x000003FF9008E838
> !j9x 0x000003FF900C0388
0x3FF900C0388 :  000003ff6e77c9c0 000003ff6e73a8a8 [ ....nw......ns.. ]
0x3FF900C0398 :  000003ff6e73ac50 000003ff6e73bd70 [ ....ns.P....ns.p ]
> !j9object 0x000003ff6e77c9c0
!J9Object 0x000003FF6E77C9C0 {
    struct J9Class* clazz = !j9class 0x3FF902D7400 // java/lang/VirtualThread
    Object flags = 0x0000000A;
    J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
    J eetop = 0x0000000000000000 (offset = 8) (java/lang/Thread)
    J tid = 0x0000000000000000 (offset = 16) (java/lang/Thread)
    Ljava/lang/String; name = !fj9object 0x0 (offset = 40) (java/lang/Thread)
    Z interrupted = 0x00000000 (offset = 152) (java/lang/Thread)
    Ljava/lang/ClassLoader; contextClassLoader = !fj9object 0x0 (offset = 48) (java/lang/Thread)
    Ljava/security/AccessControlContext; inheritedAccessControlContext = !fj9object 0x0 (offset = 56) (java/lang/Thread)
    Ljava/lang/Thread$FieldHolder; holder = !fj9object 0x0 (offset = 64) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; threadLocals = !fj9object 0x0 (offset = 72) (java/lang/Thread)
    Ljava/lang/ThreadLocal$ThreadLocalMap; inheritableThreadLocals = !fj9object 0x0 (offset = 80) (java/lang/Thread)
    Ljava/lang/Object; extentLocalBindings = !fj9object 0x0 (offset = 88) (java/lang/Thread)
    Ljava/lang/Object; interruptLock = !fj9object 0x0 (offset = 96) (java/lang/Thread)
    Ljava/lang/Object; parkBlocker = !fj9object 0x0 (offset = 104) (java/lang/Thread)
    Lsun/nio/ch/Interruptible; nioBlocker = !fj9object 0x0 (offset = 112) (java/lang/Thread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 120) (java/lang/Thread)
    Ljava/lang/Thread$UncaughtExceptionHandler; uncaughtExceptionHandler = !fj9object 0x0 (offset = 128) (java/lang/Thread)
    J threadLocalRandomSeed = 0x0000000000000000 (offset = 24) (java/lang/Thread)
    I threadLocalRandomProbe = 0x00000000 (offset = 156) (java/lang/Thread)
    I threadLocalRandomSecondarySeed = 0x00000000 (offset = 160) (java/lang/Thread)
    Ljdk/internal/vm/ThreadContainer; container = !fj9object 0x0 (offset = 136) (java/lang/Thread)
    Ljdk/internal/vm/StackableScope; headStackableScopes = !fj9object 0x0 (offset = 144) (java/lang/Thread)
    Z started = 0x00000000 (offset = 164) (java/lang/Thread)
    Z stopCalled = 0x00000000 (offset = 168) (java/lang/Thread)
    J tls = 0x0000000000000000 (offset = 32) (java/lang/Thread) <hidden>
    Ljava/util/concurrent/Executor; scheduler = !fj9object 0x0 (offset = 184) (java/lang/VirtualThread)
    Ljdk/internal/vm/Continuation; cont = !fj9object 0x0 (offset = 192) (java/lang/VirtualThread)
    Ljava/lang/Runnable; runContinuation = !fj9object 0x0 (offset = 200) (java/lang/VirtualThread)
    I state = 0x00000000 (offset = 172) (java/lang/VirtualThread)
    Z parkPermit = 0x00000000 (offset = 240) (java/lang/VirtualThread)
    Ljava/lang/Thread; carrierThread = !fj9object 0x0 (offset = 208) (java/lang/VirtualThread)
    Ljava/util/concurrent/CountDownLatch; termination = !fj9object 0x0 (offset = 216) (java/lang/VirtualThread)
    Ljava/lang/VirtualThread; linkNext = !fj9object 0x3ff8e3e3d10 (offset = 232) (java/lang/VirtualThread) <hidden>
    Ljava/lang/VirtualThread; linkPrevious = !fj9object 0x3ff8e3e3d10 (offset = 224) (java/lang/VirtualThread) <hidden>
    J inspectorCount = 0x0000000000000000 (offset = 176) (java/lang/VirtualThread) <hidden>
    I isSuspendedByJVMTI = 0x00000000 (offset = 244) (java/lang/VirtualThread) <hidden>
}
> !fj9object 0x3ff8e3e3d10
Unable to read object clazz at 0x000003FF8E3E3D10 (clazz = 0x0408040804080400)
> !j9x 0x000003FF8E3E3c80,100,4
0x3FF8E3E3C80 :  60520324 523ce052 8ba452c5 60530f44 [ `R.$R<.R..R.`S.D ]
0x3FF8E3E3C90 :  53508053 976453db a4541fe4 54642454 [ SP.S.dS..T..Td$T ]
0x3FF8E3E3CA0 :  a86454ec a45530e4 5577c455 bc045600 [ .dT..U0.Uw.U..V. ]
0x3FF8E3E3CB0 :  44564484 5688c456 cd045711 44575584 [ DVD.V..V..W.DWU. ]
0x3FF8E3E3CC0 :  5799c457 de045822 44586684 58ad6458 [ W..W..X"DXf.X.dX ]
0x3FF8E3E3CD0 :  f1a45935 e4597a24 59be645a 02a45a46 [ ..Y5.Yz$Y.dZ..ZF ]
0x3FF8E3E3CE0 :  e45a8b24 5acf645b 13a45b57 e45b9ec4 [ .Z.$Z.d[..[W.[.. ]
0x3FF8E3E3CF0 :  5bed845c 27445c76 047f0000 0d9c0408 [ [..\'D\v........ ]
0x3FF8E3E3D00 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D10 :  04080408 04080408 04080408 04080408 [ ................ ] <--- pointer mid-object
0x3FF8E3E3D20 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D30 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D40 :  04080408 04080408 04080408 04080408 [ ................ ]
0x3FF8E3E3D50 :  04080408 04080402 3e784842 ae784848 [ ........>xHB.xHH ]
0x3FF8E3E3D60 :  000003ff 90286100 0000015c 00000000 [ .....(a....\.... ]
0x3FF8E3E3D70 :  000003ff 8e3e3d78 01000000 07ffffff [ .....>=x........ ]
babsingh commented 1 year ago

@babsingh can you please take a look

In @JasonFengJ9's latest grinder: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29692, all failures are timeouts from https://github.com/eclipse-openj9/openj9/issues/16340. The GC segfault, reported in this issue, is not seen.

Corefiles for @dmitripivkine's grinder: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Grinder/29676/openjdk_test_output.tar.gz

It is not just the liveVirtualThreadList which has CORRUPT VirtualThread objects as mentioned by @dmitripivkine in https://github.com/eclipse-openj9/openj9/issues/16351#issuecomment-1324040999.

Even, the Java stack has CORRUPT VirtualThread objects; see the jdmpview output below. The CORRUPT VirtualThread is derived using Thread.currentThread() in Access.parkVirtualThread(...). Thread.currentThread() is JIT inlined; so, the JIT may be involved. fyi @tajila @0xdaryl

https://github.com/eclipse-openj9/openj9/blob/55c52ff26d9a8755b9de00704dee2b7f6543a041/jcl/src/java.base/share/classes/java/lang/Access.java#L586-L592

jdmpview output

!stack 0x3fee4064c00    !j9vmthread 0x3fee4064c00   !j9thread 0x3fee0022e18 tid 0x1db14 (121620) // (worker-1)

> info thread 121620
 process id: 121548

 no native threads found with specified id

Java threads not associated with known native threads:

    name:          <corrupt data encountered>
    id:            121620
    Thread object: <corrupt data encountered>
    Priority:      <corrupt data encountered>
    Thread.State:  WAITING
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
      waiting to be notified on: "Live virtual thread list mutex" with ID 0x3ff9008e838 owner name: <unowned>
    Java stack frames:
     bp: 0x000003fed800a528  method: void java/lang/VirtualThread.notifyJvmtiUnmountBegin(boolean)  (Native Method)
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a560  method: boolean java/lang/VirtualThread.yieldContinuation()  source: VirtualThread.java:372
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a5c0  method: void java/lang/VirtualThread.parkNanos(long)  source: VirtualThread.java:537
      objects: 0x3ff8e30d2a0 0x3ff8e30d4d8
     bp: 0x000003fed800a600  method: void java/lang/Access.parkVirtualThread(long)  source: Access.java:502
      objects: <no objects in this frame>
     bp: 0x000003fed800a628  method: void jdk/internal/misc/VirtualThreads.park(long)  source: VirtualThreads.java:67
      objects: <no objects in this frame>
     bp: 0x000003fed800a650  method: void java/util/concurrent/locks/LockSupport.parkNanos(long)  source: LockSupport.java:408
      objects: <no objects in this frame>
     bp: 0x000003fed800a688  method: void GetStackTraceALot.lambda$main$0(java.util.concurrent.atomic.AtomicInteger, int)  source: GetStackTraceALot.java:93
      objects: 0x3ff8e30d3a8
     bp: 0x000003fed800a6a8  method: void GetStackTraceALot$$Lambda$17/0x0000000095261c60.run()  source: null:<data unavailable>
      objects: <no objects in this frame>
     bp: 0x000003fed800a708  method: void java/lang/VirtualThread.run(Runnable)  source: VirtualThread.java:292
      objects: 0x3ff8e30d2a0
     bp: 0x000003fed800a730  method: void java/lang/VirtualThread$VThreadContinuation.lambda$new$0(VirtualThread, Runnable)  source: VirtualThread.java:179
      objects: <no objects in this frame>
     bp: 0x000003fed800a750  method: void java/lang/VirtualThread$VThreadContinuation$$Lambda$23/0x0000000094ea1a08.run()  source: null:<data unavailable>
      objects: <no objects in this frame>
     bp: 0x000003fed800a778  method: void jdk/internal/vm/Continuation.execute(jdk.internal.vm.Continuation)  source: Continuation.java:171
      objects: 0x3ff8e30d538

> !stackslots 0x3fee4064c00
<3fee4064c00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x000003FEE4064C00 ***
<3fee4064c00>   ITERATE_O_SLOTS
<3fee4064c00>   RECORD_BYTECODE_PC_OFFSET
<3fee4064c00> Initial values: walkSP = 0x000003FED800A4F8, PC = 0x0000000000000007, literals = 0x0000000000000000, A0 = 0x000003FED800A528, j2iFrame = 0x0000000000000000, ELS = 0x000003FF8E67EA10, decomp = 0x0000000000000000
<3fee4064c00> JNI native method frame: bp = 0x000003FED800A518, sp = 0x000003FED800A4F8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x000003FED800A528, flags = 0x0000000000000000
<3fee4064c00>   Method: java/lang/VirtualThread.notifyJvmtiUnmountBegin(Z)V !j9method 0x000003FF902D6FA0
<3fee4064c00>   Using signature mapper
<3fee4064c00>   Arguments starting at 0x000003FED800A528 for UDATA(0x0000000000000002) slots
<3fee4064c00>       O-Slot: a0[0x000003FED800A528] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00>       I-Slot: a1[0x000003FED800A520] = 0x0000000000000000
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A540, sp = 0x000003FED800A530, pc = 0x000003FF1D4EB36A, cp = 0x000003FF902D5910, arg0EA = 0x000003FED800A560, flags = 0x0000000000000000
<3fee4064c00>   Method: java/lang/VirtualThread.yieldContinuation()Z !j9method 0x000003FF902D6B00
<3fee4064c00>   Bytecode index = 10
<3fee4064c00>   Using local mapper
<3fee4064c00>   Locals starting at 0x000003FED800A560 for 0x0000000000000004 slots
<3fee4064c00>       O-Slot: a0[0x000003FED800A560] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00>       I-Slot: t1[0x000003FED800A558] = 0x00000001D800A562
<3fee4064c00>       I-Slot: t2[0x000003FED800A550] = 0x000003FF1D4EB6FC
<3fee4064c00>       I-Slot: t3[0x000003FED800A548] = 0x000003FF902D6BE0
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A578, sp = 0x000003FED800A568, pc = 0x000003FF1D4EB6FC, cp = 0x000003FF902D5910, arg0EA = 0x000003FED800A5C0, flags = 0x0000000000000000
<3fee4064c00>   Method: java/lang/VirtualThread.parkNanos(J)V !j9method 0x000003FF902D6BE0
<3fee4064c00>   Bytecode index = 60
<3fee4064c00>   Using local mapper
<3fee4064c00>   Locals starting at 0x000003FED800A5C0 for 0x0000000000000009 slots
<3fee4064c00>       O-Slot: a0[0x000003FED800A5C0] = 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
<3fee4064c00>       I-Slot: a1[0x000003FED800A5B8] = 0x000003FEE404FBD2
<3fee4064c00>       I-Slot: a2[0x000003FED800A5B0] = 0x000000000007A120
<3fee4064c00>       I-Slot: t3[0x000003FED800A5A8] = 0x000003FEE404FBD2
<3fee4064c00>       I-Slot: t4[0x000003FED800A5A0] = 0x1729C1C46038E12E
<3fee4064c00>       I-Slot: t5[0x000003FED800A598] = 0x000003FF902FADB0
<3fee4064c00>       O-Slot: t6[0x000003FED800A590] = 0x000003FF8E30D4D8
<3fee4064c00>       I-Slot: t7[0x000003FED800A588] = 0x000003FF1D4A1501
<3fee4064c00>       I-Slot: t8[0x000003FED800A580] = 0x000003FF902A9200
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A5D8, sp = 0x000003FED800A5C8, pc = 0x000003FF1D5248A1, cp = 0x000003FF902FAEB0, arg0EA = 0x000003FED800A600, flags = 0x0000000000000000
<3fee4064c00>   Method: java/lang/Access.parkVirtualThread(J)V !j9method 0x000003FF902FADB0
<3fee4064c00>   Bytecode index = 21
<3fee4064c00>   Using local mapper
<3fee4064c00>   Locals starting at 0x000003FED800A600 for 0x0000000000000005 slots
<3fee4064c00>       I-Slot: a0[0x000003FED800A600] = 0x000003FF6E5F3890
<3fee4064c00>       I-Slot: a1[0x000003FED800A5F8] = 0x000003FF1DEF4BA4
<3fee4064c00>       I-Slot: a2[0x000003FED800A5F0] = 0x000000000007A120
<3fee4064c00>       I-Slot: t3[0x000003FED800A5E8] = 0x000003FF8E443B68
<3fee4064c00>       I-Slot: t4[0x000003FED800A5E0] = 0x000003FF8E443B68
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A618, sp = 0x000003FED800A608, pc = 0x000003FF1DEF4BAA, cp = 0x000003FEE4057FC0, arg0EA = 0x000003FED800A628, flags = 0x0000000000000000
<3fee4064c00>   Method: jdk/internal/misc/VirtualThreads.park(J)V !j9method 0x000003FEE4057D30
<3fee4064c00>   Bytecode index = 6
<3fee4064c00>   Using local mapper
<3fee4064c00>   Locals starting at 0x000003FED800A628 for 0x0000000000000002 slots
<3fee4064c00>       I-Slot: a0[0x000003FED800A628] = 0x0000000100000008
<3fee4064c00>       I-Slot: a1[0x000003FED800A620] = 0x000000000007A120
<3fee4064c00> Bytecode frame: bp = 0x000003FED800A640, sp = 0x000003FED800A630, pc = 0x000003FF1D4E2430, cp = 0x000003FF902CEA70, arg0EA = 0x000003FED800A650, flags = 0x0000000000000000
<3fee4064c00>   Method: java/util/concurrent/locks/LockSupport.parkNanos(J)V !j9method 0x000003FF902CE9E8
<3fee4064c00>   Bytecode index = 16
<3fee4064c00>   Using local mapper
<3fee4064c00>   Locals starting at 0x000003FED800A650 for 0x0000000000000002 slots
<3fee4064c00>       I-Slot: a0[0x000003FED800A650] = 0x00000001E404FC58
<3fee4064c00>       I-Slot: a1[0x000003FED800A648] = 0x000000000007A120

> !j9object 0x000003FF8E30D2A0   <---- CORRUPT VirtualThread 
Unable to read object clazz at 0x000003FF8E30D2A0 (clazz = 0x000003FF6E777300)

> !j9x 0x000003FF8E30D2A0-0x50,0x100
0x3FF8E30D250 :  000003ff6e65f62c 0000000000000000 [ ....ne.,........ ]
0x3FF8E30D260 :  000003fed404e0d0 000003fed4088398 [ ................ ]
0x3FF8E30D270 :  000003ff6e749640 000003ff8e310570 [ ....nt.@.....1.p ]
0x3FF8E30D280 :  000003ff8e30d218 0000000000000000 [ .....0.......... ]
0x3FF8E30D290 :  0000000000000000 0601100a00000000 [ ................ ]
0x3FF8E30D2A0 :  000003ff6e7773ec 0000000000000000 [ ....nws......... ] <--- CORRUPT VirtualThread
0x3FF8E30D2B0 :  0000000000000000 000000000000001c [ ................ ]
0x3FF8E30D2C0 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D2D0 :  000003ff6e58b308 000003ff6e5f2d20 [ ....nX......n_-  ]
0x3FF8E30D2E0 :  000003ff6e6ca908 0000000000000000 [ ....nl.......... ]
0x3FF8E30D2F0 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D300 :  0000000000000000 000003ff8e310590 [ .............1.. ]
0x3FF8E30D310 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D320 :  0000000000000000 0000000000000000 [ ................ ]
0x3FF8E30D330 :  000003ff8e3001f0 0000000000000000 [ .....0.......... ]
0x3FF8E30D340 :  0000000000000000 0000000000000000 [ ................ ]
JasonFengJ9 commented 1 year ago

Results from 50x grinder -Xtrace:maximal={j9scar{level5}},buffers=2m: Iteration 0

12:06:44  17:06:34.402 0x70ba3800a400    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk19/jdk19-linux-ppc64le-openj9/workspace/build/src/omr/gc/base/standard/Scavenger.cpp:1657: ((false))

Iteration 1 - timed out Iteration 2 - passed Iteration 3 - passed Iteration 4 - no failure though the job status UNSTABLE

tajila commented 1 year ago

50x grinder with assertions https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29825

babsingh commented 1 year ago

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29799/

@tajila Unzipping files failed in the grinder.

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/62/ - no diagnostics

dmitripivkine commented 1 year ago

another failure https://github.com/eclipse-openj9/openj9/issues/16249#issuecomment-1332775416 (single element Virtual Threads List next/previous stale pointer)

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/61/ jdk_lang_1 java/lang/Thread/virtual/ParkWithFixedThreadPool.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/61/openjdk_test_output.tar.gz

00:16:44  Type=Segmentation error vmState=0x0002000f
00:16:44  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
00:16:44  Handler1=09001000A1017FB0 Handler2=09001000A0F57020
00:16:44  R0=321825CB00000000 R1=0000010022A01760 R2=09001000A0F26978 R3=0000010010209150
00:16:44  R4=0000010022A0B3D8 R5=0000010022A01B10 R6=0000010022A018B0 R7=0000010022A018A8
00:16:44  R8=0000010022A018A0 R9=000001001038F450 R10=0000000000000000 R11=0A0000001FF84A80
00:16:44  R12=FFFFFFFF82000022 R13=0000010022A0C800 R14=0000000000000001 R15=0A0000001FD90E48
00:16:44  R16=0A0000001FF9A390 R17=0A0000001FF9A390 R18=000001001038F370 R19=0000000000000000
00:16:44  R20=0000000000000000 R21=0A0000001FD91CB0 R22=000001001038F348 R23=0000000000002C20
00:16:44  R24=0000000000002C18 R25=00000100102091D8 R26=321825CB00000000 R27=0000000000000000
00:16:44  R28=0000010022A018A0 R29=0000010022A018A8 R30=0000010022A018B0 R31=0000010022A01B10
00:16:44  IAR=090000001033D364 LR=09000000104038B8 MSR=A00000000200D032 CTR=090000001042C424
00:16:44  CR=2400002200000004 FPSCR=8202000000000000 XER=0000000482020000
00:16:44  FPR0 000000000000004c (f: 76.000000, d: 3.754899e-322)
00:16:44  FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
00:16:44  FPR2 3fc7344ba27d52af (f: 2726122240.000000, d: 1.812834e-01)
00:16:44  FPR3 3f51261904000000 (f: 67108864.000000, d: 1.046681e-03)
00:16:44  FPR4 bf90ecf58ddc7f4a (f: 2380037888.000000, d: -1.652893e-02)
00:16:44  FPR5 40a6d60000000000 (f: 0.000000, d: 2.923000e+03)
00:16:44  FPR6 4070b1f20d4afc0c (f: 223018000.000000, d: 2.671216e+02)
00:16:44  FPR7 bfb62e42fefa4000 (f: 4277813248.000000, d: -8.664340e-02)
00:16:44  FPR8 3f6069c0ff9e20ca (f: 4288553216.000000, d: 2.003552e-03)
00:16:44  FPR9 402bb9d3beb8d000 (f: 3199782912.000000, d: 1.386294e+01)
00:16:44  FPR10 408f400000000000 (f: 0.000000, d: 1.000000e+03)
00:16:44  FPR11 43300000000003e8 (f: 1000.000000, d: 4.503600e+15)
00:16:44  FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
00:16:44  FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:16:44  Module=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
00:16:44  Module_base_address=09000000102AF000
00:16:44  Target=2_90_20221201_130 (AIX 7.1)
00:16:44  CPU=ppc64 (16 logical CPUs) (0x780000000 RAM)
00:16:44  ----------- Stack Backtrace -----------
00:16:44  _ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0xb4 (0x09000000104038B8 [libj9gc_full29.so+0x1548b8])
00:16:44  _ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x1170 (0x090000001040DCF4 [libj9gc_full29.so+0x15ecf4])
00:16:44  _ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x2f0 (0x090000001040EAB4 [libj9gc_full29.so+0x15fab4])
00:16:44  _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x388 (0x090000001040EF4C [libj9gc_full29.so+0x15ff4c])
00:16:44  _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x10 (0x0900000010444694 [libj9gc_full29.so+0x195694])
00:16:44  _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x1d4 (0x090000001049DAD8 [libj9gc_full29.so+0x1eead8])
00:16:44  _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x1d8 (0x090000001049D7DC [libj9gc_full29.so+0x1ee7dc])
00:16:44  omrsig_protect+0x4fc (0x090000000E24E1E0 [libj9prt29.so+0x5f1e0])
00:16:44  dispatcher_thread_proc+0x50 (0x090000001049D8B4 [libj9gc_full29.so+0x1ee8b4])
00:16:44  thread_wrapper+0x558 (0x090000000E2B799C [libj9thr29.so+0x599c])
00:16:44  _pthread_body+0xf0 (0x090000000051FE14 [libpthreads.a+0x3e14])
00:16:44  ---------------------------------------
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/61 jdk_lang_1 java/lang/Thread/virtual/stress/GetStackTraceALot.java#id0 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_ppc64le_linux_Nightly/61/openjdk_test_output.tar.gz

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_x86-64_mac_Nightly/60 jdk_lang_1 java/lang/Thread/virtual/ParkWithFixedThreadPool.java No diagnostics.