Closed pshipton closed 2 months ago
@hzongaro can someone take a look at this with priority pls. It seems it may be responsible for taking AIX machines offline due to the amount of output. There is 800MB of console output from the backtraces, which I assume is blocking capturing any core file.
~Excluding the AIX tests https://github.com/adoptium/aqa-tests/pull/5467~
See https://github.com/eclipse-openj9/openj9/issues/19947#issuecomment-2261119303
Also on zlinux
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_s390x_linux_Personal_testList_0/178/
MathLoadTest_all_special_5m_17 -Xcompressedrefs -Xgcpolicy:balanced -Xquickstart -Xjit
16:42:01 MLT stderr Type=Segmentation error vmState=0x000509ff
16:42:01 MLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=f0e7d270 Signal_Code=00000001
16:42:01 MLT stderr Handler1=000003FF99649538 Handler2=000003FF99531A60 InaccessibleAddress=0000000000000000
16:42:01 MLT stderr gpr0=000003FF345218C0 gpr1=000003FF8D28C068 gpr2=0000000000000000 gpr3=0000000000000000
16:42:01 MLT stderr gpr4=000003FF8D4FCDDE gpr5=000003FF34420D40 gpr6=000003FF34420380 gpr7=000003FF34420A80
16:42:01 MLT stderr gpr8=000003FF34420F48 gpr9=000003FF345217D0 gpr10=000003FF34420CE0 gpr11=000003FF0000000B
16:42:01 MLT stderr gpr12=000003FF8DA05000 gpr13=0000000000000000 gpr14=000003FF8D4FCE10 gpr15=000003FF6E9F3F80
16:42:01 MLT stderr psw=000003FF8D4F5794 mask=0705200180000000 fpc=0008fe00 bea=000003FF8D4FCE0A
16:42:01 MLT stderr fpr0=3f40000000000000 (f: 0.000000, d: 4.882812e-04)
16:42:01 MLT stderr fpr1=3fb93d4bb7e327a9 (f: 3085117440.000000, d: 9.859155e-02)
16:42:01 MLT stderr fpr2=4160000000000000 (f: 0.000000, d: 8.388608e+06)
16:42:01 MLT stderr fpr3=3fd999f1abd2dada (f: 2882722560.000000, d: 4.000210e-01)
16:42:01 MLT stderr fpr4=4480000000000000 (f: 0.000000, d: 9.444733e+21)
16:42:01 MLT stderr fpr5=3fcc723e7dcde3fb (f: 2110645248.000000, d: 2.222365e-01)
16:42:01 MLT stderr fpr6=3fe0000000000000 (f: 0.000000, d: 5.000000e-01)
16:42:01 MLT stderr fpr7=3f83e83bb8b32a19 (f: 3098749440.000000, d: 9.720294e-03)
16:42:01 MLT stderr fpr8=000003ff34520d50 (f: 877792576.000000, d: 2.171235e-311)
16:42:01 MLT stderr fpr9=000003ff400071f0 (f: 1073771008.000000, d: 2.171332e-311)
16:42:01 MLT stderr fpr10=000003ff34521090 (f: 877793408.000000, d: 2.171235e-311)
16:42:01 MLT stderr fpr11=0000000000cc6480 (f: 13395072.000000, d: 6.618045e-317)
16:42:01 MLT stderr fpr12=0000000000063008 (f: 405512.000000, d: 2.003495e-318)
16:42:01 MLT stderr fpr13=0000000000000001 (f: 1.000000, d: 4.940656e-324)
16:42:01 MLT stderr fpr14=000003ffc1277e0c (f: 3240590848.000000, d: 2.172403e-311)
16:42:01 MLT stderr fpr15=0000000000cc6470 (f: 13395056.000000, d: 6.618037e-317)
16:42:01 MLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_s390x_linux_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
16:42:01 MLT stderr Module_base_address=000003FF8CB00000
16:42:01 MLT stderr
16:42:01 MLT stderr Method_being_compiled=org/junit/runner/Result$Listener.testRunStarted(Lorg/junit/runner/Description;)V
16:42:01 MLT stderr Target=2_90_20240727_871 (Linux 3.10.0-1160.118.1.el7.s390x)
16:42:01 MLT stderr CPU=s390x (4 logical CPUs) (0x1ec1b1000 RAM)
16:42:01 MLT stderr ----------- Stack Backtrace -----------
16:42:01 MLT stderr _ZN18TR_RegionStructure18collapseIntoParentEv+0xc (0x000003FF8D4F5794 [libj9jit29.so+0x9f5794])
16:42:01 MLT stderr _ZN18TR_RegionStructure7addEdgeEPN2TR7CFGEdgeEb+0x2c0 (0x000003FF8D4FCE10 [libj9jit29.so+0x9fce10])
16:42:01 MLT stderr _ZN3OMR3CFG7addEdgeEPN2TR7CFGEdgeE+0x8e (0x000003FF8D21E106 [libj9jit29.so+0x71e106])
16:42:01 MLT stderr _ZN18TR_J9InlinerPolicy22genCodeForUnsafeGetPutEPN2TR4NodeES2_NS0_8DataTypeEPNS0_7TreeTopES5_PNS0_15SymbolReferenceES5_S5_S5_S5_bbbbbS2_+0xaec (0x000003FF8CF1D05C [libj9jit29.so+0x41d05c])
16:42:01 MLT stderr _ZN18TR_J9InlinerPolicy25createUnsafePutWithOffsetEPN2TR20ResolvedMethodSymbolES2_PNS0_7TreeTopEPNS0_4NodeENS0_8DataTypeEbbb+0x828 (0x000003FF8CF1E158 [libj9jit29.so+0x41e158])
16:42:01 MLT stderr _ZN18TR_J9InlinerPolicy16inlineUnsafeCallEPN2TR20ResolvedMethodSymbolES2_PNS0_7TreeTopEPNS0_4NodeE+0x2be (0x000003FF8CF217DE [libj9jit29.so+0x4217de])
16:42:01 MLT stderr _ZN18TR_J9InlinerPolicy24tryToInlineTrivialMethodEP12TR_CallStackP13TR_CallTarget+0x1a4 (0x000003FF8CF22894 [libj9jit29.so+0x422894])
16:42:01 MLT stderr _ZN14TR_InlinerBase17inlineCallTarget2EP12TR_CallStackP13TR_CallTargetPPN2TR7TreeTopEbi+0x1ba (0x000003FF8D2DE25A [libj9jit29.so+0x7de25a])
16:42:01 MLT stderr _ZN14TR_InlinerBase16inlineCallTargetEP12TR_CallStackP13TR_CallTargetbP14TR_PrexArgInfoPPN2TR7TreeTopE+0x1a6 (0x000003FF8CF526AE [libj9jit29.so+0x4526ae])
16:42:01 MLT stderr _ZN13TR_InlineCall10inlineCallEPN2TR7TreeTopEP19TR_OpaqueClassBlockbP14TR_PrexArgInfoi+0x3d6 (0x000003FF8D2E1ABE [libj9jit29.so+0x7e1abe])
16:42:01 MLT stderr _ZN3OMR16ValuePropagation24doDelayedTransformationsEv+0x862 (0x000003FF8D5A83B2 [libj9jit29.so+0xaa83b2])
16:42:01 MLT stderr _ZN2TR21LocalValuePropagation19postPerformOnBlocksEv+0x1e (0x000003FF8D5BDB4E [libj9jit29.so+0xabdb4e])
16:42:01 MLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii.localalias+0x8b6 (0x000003FF8D42382E [libj9jit29.so+0x92382e])
16:42:01 MLT stderr _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii.localalias+0xf44 (0x000003FF8D423EBC [libj9jit29.so+0x923ebc])
16:42:01 MLT stderr _ZN3OMR9Optimizer8optimizeEv+0x1e6 (0x000003FF8D4255F6 [libj9jit29.so+0x9255f6])
16:42:01 MLT stderr _ZN3OMR11Compilation7compileEv+0xc54 (0x000003FF8D1BCB1C [libj9jit29.so+0x6bcb1c])
16:42:01 MLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4de (0x000003FF8CC95A06 [libj9jit29.so+0x195a06])
16:42:01 MLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x3b0 (0x000003FF8CC96B30 [libj9jit29.so+0x196b30])
16:42:01 MLT stderr omrsig_protect+0x3d8 (0x000003FF99532BB0 [libj9prt29.so+0x32bb0])
16:42:01 MLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x402 (0x000003FF8CC94192 [libj9jit29.so+0x194192])
16:42:01 MLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x13c (0x000003FF8CC9471C [libj9jit29.so+0x19471c])
16:42:01 MLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x402 (0x000003FF8CC931A2 [libj9jit29.so+0x1931a2])
16:42:01 MLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF8CC93694 [libj9jit29.so+0x193694])
16:42:01 MLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF8CC9372C [libj9jit29.so+0x19372c])
16:42:01 MLT stderr omrsig_protect+0x3d8 (0x000003FF99532BB0 [libj9prt29.so+0x32bb0])
16:42:01 MLT stderr _Z21compilationThreadProcPv+0x196 (0x000003FF8CC93B6E [libj9jit29.so+0x193b6e])
16:42:01 MLT stderr thread_wrapper+0xf6 (0x000003FF99488A46 [libj9thr29.so+0x8a46])
16:42:01 MLT stderr start_thread+0xea (0x000003FF9A008312 [libpthread.so.0+0x8312])
16:42:01 MLT stderr (0x000003FF99E0E232 [libc.so.6+0x10e232])
16:42:01 MLT stderr ---------------------------------------
17:48:17 MiniMix_3h_0 17:48:17 MathLoadTest_bigdecimal_special_5m_12
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_s390x_linux_Personal_testList_1/178/ 18:09:55 DaaLoadTest_daa1_special_5m_12 18:09:55 DaaLoadTest_daa2_special_5m_12 18:09:55 DaaLoadTest_daa2_special_5m_17 18:09:55 MathLoadTest_bigdecimal_special_5m_13
https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_s390x_linux_Personal_testList_3/122 https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_s390x_linux_Personal/181 https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_s390x_linux_Personal/141 https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_s390x_linux_Personal/55/ etc.
For the time being I'm disabling the special.system test runs on AIX and zlinux. I've also added back x86_mac, aarch64_mac and xlinux, which were previously disabled due to infra problems.
@midronij, is it possible this is related to your recent changes from pull request #17969? Looking at the jitdump from https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_special.system_s390x_linux_Personal_testList_0/178/system_test_output.tar.gz, it looks like the crash is happening while processing a call to Unsafe.putLongVolatile
for an instance of AtomicLong
.
https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_linux_Nightly_testList_0/25/ jdk_lang_j9_0 java/lang/Class/getDeclaredField/FieldSetAccessibleTest.java
04:47:21 Type=Segmentation error vmState=0x000509ff
04:47:21 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
04:47:21 Handler1=00007FA1EF492F90 Handler2=00007FA1F09B7740 InaccessibleAddress=0000000000000088
04:47:21 RDI=0000000000000000 RSI=0000000000000000 RAX=0000000000000000 RBX=0000000000000000
04:47:21 RCX=00007FA1B01B2901 RDX=00007FA1B01AE850 R8=0000000000000018 R9=00007FA1EC3A9730
04:47:21 R10=00007FA1B011F040 R11=00007FA1F008DD50 R12=0000000000000000 R13=00007FA1B01B23B0
04:47:21 R14=0000000000000054 R15=00007FA1B0177BE0
04:47:21 RIP=00007FA1EE9E39E4 GS=0000 FS=0000 RSP=00007FA1EC3A96E0
04:47:21 EFlags=0000000000010202 CS=0033 RBP=00007FA1B01AE850 ERR=0000000000000004
04:47:21 TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000088
04:47:21 xmm0=00007fa1b01b1da0 (f: 2954567168.000000, d: 6.933343e-310)
04:47:21 xmm1=00007fa1b01b1da0 (f: 2954567168.000000, d: 6.933343e-310)
04:47:21 xmm2=00007fa1b0177cd0 (f: 2954329344.000000, d: 6.933343e-310)
04:47:21 xmm3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:47:21 xmm4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:47:21 xmm5=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:47:21 xmm6=00007fa1e82ae630 (f: 3895125504.000000, d: 6.933389e-310)
04:47:21 xmm7=00007fa1b0176420 (f: 2954322944.000000, d: 6.933343e-310)
04:47:21 xmm8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:47:21 xmm9=000727f90000fd22 (f: 64802.000000, d: 9.951842e-309)
04:47:21 xmm10=ff000000000000ff (f: 255.000000, d: -5.486124e+303)
04:47:21 xmm11=0000015200000151 (f: 337.000000, d: 7.172346e-312)
04:47:21 xmm12=0000013d00000140 (f: 320.000000, d: 6.726727e-312)
04:47:21 xmm13=000001380000013f (f: 319.000000, d: 6.620627e-312)
04:47:21 xmm14=0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
04:47:21 xmm15=000001420000013b (f: 315.000000, d: 6.832826e-312)
04:47:21 Module=/home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_x86-64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
04:47:21 Module_base_address=00007FA1EE176000
04:47:21
04:47:21 Method_being_compiled=FieldSetAccessibleTest.test(Ljava/lang/Iterable;)V
04:47:21 Target=2_90_20240801_26 (Linux 3.10.0-1160.114.2.el7.x86_64)
04:47:21 CPU=amd64 (4 logical CPUs) (0x1e8cbc000 RAM)
04:47:21 ----------- Stack Backtrace -----------
04:47:21 _ZN18TR_RegionStructure18collapseIntoParentEv+0x14 (0x00007FA1EE9E39E4 [libj9jit29.so+0x86d9e4])
04:47:21 _ZN18TR_RegionStructure7addEdgeEPN2TR7CFGEdgeEb+0x2e2 (0x00007FA1EE9E9502 [libj9jit29.so+0x873502])
04:47:21 _ZN3OMR3CFG7addEdgeEPN2TR7CFGEdgeE+0x62 (0x00007FA1EE76E722 [libj9jit29.so+0x5f8722])
04:47:21 _ZN18TR_J9InlinerPolicy22genCodeForUnsafeGetPutEPN2TR4NodeES2_NS0_8DataTypeEPNS0_7TreeTopES5_PNS0_15SymbolReferenceES5_S5_S5_S5_bbbbbS2_+0xd33 (0x00007FA1EE51ADD3 [libj9jit29.so+0x3a4dd3])
04:47:21 _ZN18TR_J9InlinerPolicy25createUnsafePutWithOffsetEPN2TR20ResolvedMethodSymbolES2_PNS0_7TreeTopEPNS0_4NodeENS0_8DataTypeEbbb+0x5ee (0x00007FA1EE51B8BE [libj9jit29.so+0x3a58be])
04:47:21 _ZN18TR_J9InlinerPolicy16inlineUnsafeCallEPN2TR20ResolvedMethodSymbolES2_PNS0_7TreeTopEPNS0_4NodeE+0x255 (0x00007FA1EE51ED35 [libj9jit29.so+0x3a8d35])
04:47:21 _ZN18TR_J9InlinerPolicy24tryToInlineTrivialMethodEP12TR_CallStackP13TR_CallTarget+0x166 (0x00007FA1EE51FAE6 [libj9jit29.so+0x3a9ae6])
04:47:21 _ZN14TR_InlinerBase17inlineCallTarget2EP12TR_CallStackP13TR_CallTargetPPN2TR7TreeTopEbi+0x11b (0x00007FA1EE813ACB [libj9jit29.so+0x69dacb])
04:47:21 _ZN14TR_InlinerBase16inlineCallTargetEP12TR_CallStackP13TR_CallTargetbP14TR_PrexArgInfoPPN2TR7TreeTopE+0x1a3 (0x00007FA1EE5495E3 [libj9jit29.so+0x3d35e3])
04:47:21 _ZN13TR_InlineCall10inlineCallEPN2TR7TreeTopEP19TR_OpaqueClassBlockbP14TR_PrexArgInfoi+0x35e (0x00007FA1EE81705E [libj9jit29.so+0x6a105e])
04:47:21 _ZN3OMR16ValuePropagation24doDelayedTransformationsEv+0x973 (0x00007FA1EEA83533 [libj9jit29.so+0x90d533])
04:47:21 _ZN2TR21LocalValuePropagation19postPerformOnBlocksEv+0xa (0x00007FA1EEA95A5A [libj9jit29.so+0x91fa5a])
04:47:21 _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii.localalias+0x861 (0x00007FA1EE92D471 [libj9jit29.so+0x7b7471])
04:47:21 _ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii.localalias+0xc1f (0x00007FA1EE92D82F [libj9jit29.so+0x7b782f])
04:47:21 _ZN3OMR9Optimizer8optimizeEv+0x1b3 (0x00007FA1EE92F183 [libj9jit29.so+0x7b9183])
04:47:21 _ZN3OMR11Compilation7compileEv+0xa25 (0x00007FA1EE71EC15 [libj9jit29.so+0x5a8c15])
04:47:21 _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4bf (0x00007FA1EE2EFE7F [libj9jit29.so+0x179e7f])
04:47:21 _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x381 (0x00007FA1EE2F0EB1 [libj9jit29.so+0x17aeb1])
04:47:21 omrsig_protect+0x239 (0x00007FA1F09B83C9 [libj9prt29.so+0x2a3c9])
04:47:21 _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x385 (0x00007FA1EE2EEA15 [libj9jit29.so+0x178a15])
04:47:21 _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x128 (0x00007FA1EE2EED38 [libj9jit29.so+0x178d38])
04:47:21 _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x377 (0x00007FA1EE2EDC77 [libj9jit29.so+0x177c77])
04:47:21 _ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x00007FA1EE2EDFD2 [libj9jit29.so+0x177fd2])
04:47:21 _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x00007FA1EE2EE082 [libj9jit29.so+0x178082])
04:47:21 omrsig_protect+0x239 (0x00007FA1F09B83C9 [libj9prt29.so+0x2a3c9])
04:47:21 _Z21compilationThreadProcPv+0x17b (0x00007FA1EE2EE44B [libj9jit29.so+0x17844b])
04:47:21 thread_wrapper+0x163 (0x00007FA1F09803A3 [libj9thr29.so+0xb3a3])
04:47:21 start_thread+0xc5 (0x00007FA1F02D5EA5 [libpthread.so.0+0x7ea5])
04:47:21 clone+0x6d (0x00007FA1EFFFEB0D [libc.so.6+0xfeb0d])
04:47:21 ---------------------------------------
The tests aren't really excluded, but I've set test excluded
as a reminder to re-enable special.system testing once this is resolved.
Although the open testing is limited, the tests are failing on all platforms except for aarch64.
Reverted the suspected change causing the problem.
@pshipton I believe this contribution (https://github.com/eclipse-openj9/openj9/pull/20090) should fix this issue
Related to https://github.ibm.com/runtimes/infrastructure/issues/9201
vmState [0x514ff]: {J9VMSTATE_JIT} {globalValuePropagation}
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64_aix_Personal_testList_1/133/ DaaLoadTest_daa1_special_5m_11 Mode351
-Xgcpolicy:metronome -Xcompressedrefs
vmState [0x509ff]: {J9VMSTATE_JIT} {localValuePropagation}
DaaLoadTest_daa2_special_5m_11
There are more failures on AIX, all versions. They are hard to look at since the output is so big.
https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_ppc64_aix_Personal/122/ https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_ppc64_aix_Personal/148/ https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64_aix_Personal/133/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_ppc64_aix_Personal/48/