search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.29k stars 722 forks source link

cmdLineTester_criu_jitserverPostRestore zLinux assertion JITServerPersistentCHTable.cpp:172 #17474

Open JasonFengJ9 opened 1 year ago

JasonFengJ9 commented 1 year ago

Failure link

From an internal build(rhel7s390x-4-1):

java version "11.0.20" 2023-07-18
IBM Semeru Runtime Certified Edition 11.0.20+2 (build 11.0.20+2)
Eclipse OpenJ9 VM 11.0.20+2 (build master-3b029b0af, JRE 11 Linux s390x-64-Bit Compressed References 20230529_567 (JIT enabled, AOT enabled)
OpenJ9   - 3b029b0af
OMR      - f89af3519
JCL      - b111539de8 based on jdk-11.0.20+2)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2023-05-30T07:45:59.292Z] variation: -Xjit:vlog=vlog
[2023-05-30T07:45:59.292Z] JVM_OPTIONS:  -Xjit:vlog=vlog 

[2023-05-30T07:46:19.061Z] Testing: Test -Xnoaot
[2023-05-30T07:46:19.061Z] Test start time: 2023/05/30 00:46:18 Pacific Standard Time
[2023-05-30T07:46:19.061Z] Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false
[2023-05-30T07:46:19.061Z] Time spent starting: 14 milliseconds
[2023-05-30T07:46:36.861Z] Time spent executing: 15909 milliseconds
[2023-05-30T07:46:36.862Z] Test result: FAILED
[2023-05-30T07:46:36.862Z] Output from test:
[2023-05-30T07:46:36.862Z]  [OUT] start running script
[2023-05-30T07:46:36.862Z]  [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
[2023-05-30T07:46:36.862Z]  [OUT] export LD_BIND_NOT=on
[2023-05-30T07:46:36.862Z]  [OUT] Starting /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=51945
[2023-05-30T07:46:36.862Z]  [OUT]  9467 ?        00:00:00 jitserver
[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER EXISTS
[2023-05-30T07:46:36.862Z]  [OUT] Pre-checkpoint
[2023-05-30T07:46:36.862Z]  [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Tue May 30 00:46:20 PDT 2023, System.currentTimeMillis(): 1685432780893, System.nanoTime(): 1685432660688976596
[2023-05-30T07:46:36.862Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2023-05-30T07:46:36.862Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2023-05-30T07:46:36.862Z]  [OUT] Post-checkpoint
[2023-05-30T07:46:36.862Z]  [OUT] Removed testOutput file
[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER NO LONGER EXISTS
[2023-05-30T07:46:36.862Z]  [OUT] Terminating /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=51945
[2023-05-30T07:46:36.862Z]  [OUT] finished script
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is ready to accept incoming requests
[2023-05-30T07:46:36.862Z]  [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 88:  9547 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
[2023-05-30T07:46:36.862Z]  [ERR] Assertion failed at ../../../../../openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
[2023-05-30T07:46:36.862Z]  [ERR]   subclass info cannot be null: ensure subclasses are loaded before superclass
[2023-05-30T07:46:36.862Z]  [ERR] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa494e6) [0x3ffa91494e6]
[2023-05-30T07:46:36.862Z]  [ERR] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa56e80) [0x3ffa9156e80]
[2023-05-30T07:46:36.862Z]  [ERR] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c0778) [0x3ffa8dc0778]
[2023-05-30T07:46:36.862Z]  [ERR] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c23de) [0x3ffa8dc23de]
[2023-05-30T07:46:36.862Z]  [ERR] #4: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2a95ea) [0x3ffa89a95ea]
[2023-05-30T07:46:36.862Z]  [ERR] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2a9a78) [0x3ffa89a9a78]
[2023-05-30T07:46:36.862Z]  [ERR] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x200496) [0x3ffa8900496]
[2023-05-30T07:46:36.862Z]  [ERR] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183776) [0x3ffa8883776]
[2023-05-30T07:46:36.862Z]  [ERR] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183c66) [0x3ffa8883c66]
[2023-05-30T07:46:36.862Z]  [ERR] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183d4a) [0x3ffa8883d4a]
[2023-05-30T07:46:36.862Z]  [ERR] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34546) [0x3ffb0134546]
[2023-05-30T07:46:36.862Z]  [ERR] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1841e6) [0x3ffa88841e6]
[2023-05-30T07:46:36.862Z]  [ERR] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffb0085a14]
[2023-05-30T07:46:36.862Z]  [ERR] #13: /lib64/libpthread.so.0(+0x8312) [0x3ffb0a88312]
[2023-05-30T07:46:36.862Z]  [ERR] #14: /lib64/libc.so.6(+0x10e232) [0x3ffb098e232]
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] Unhandled exception
[2023-05-30T07:46:36.862Z]  [ERR] Type=Unhandled trap vmState=0x00000000
[2023-05-30T07:46:36.862Z]  [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
[2023-05-30T07:46:36.862Z]  [ERR] Handler1=000003FFB024CBA8 Handler2=000003FFB01333D8
[2023-05-30T07:46:36.862Z]  [ERR] gpr0=0000000000000005 gpr1=000003FF8E17B910 gpr2=0000000000000000 gpr3=0000000000002502
[2023-05-30T07:46:36.862Z]  [ERR] gpr4=0000000000000005 gpr5=000003FF000024FB gpr6=000003FF8E17A110 gpr7=000003FF7C0104A0
[2023-05-30T07:46:36.862Z]  [ERR] gpr8=000003FEF4A0FEC0 gpr9=000003FF7C012470 gpr10=000000000000053E gpr11=000003FFA959FAF0
[2023-05-30T07:46:36.862Z]  [ERR] gpr12=000003FFB0A99000 gpr13=000003FFA92F4418 gpr14=000003FFA8DC2386 gpr15=000003FF8E179FD0
[2023-05-30T07:46:36.862Z]  [ERR] psw=000003FFB0A90BC0 mask=0705200180000000 fpc=0008fe00 bea=000003FFB08D81E2
[2023-05-30T07:46:36.862Z]  [ERR] fpr0 4008000000000000 (f: 0.000000, d: 3.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr1 3fff000000000000 (f: 0.000000, d: 1.937500e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr4 4008000000000000 (f: 0.000000, d: 3.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr5 00000000a000c000 (f: 2684403712.000000, d: 1.326272e-314)
[2023-05-30T07:46:36.862Z]  [ERR] fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr7 3e3a35f300000000 (f: 0.000000, d: 6.102663e-09)
[2023-05-30T07:46:36.862Z]  [ERR] fpr8 000003fff3efae18 (f: 4092571136.000000, d: 2.172824e-311)
[2023-05-30T07:46:36.862Z]  [ERR] fpr9 000000000014feb0 (f: 1375920.000000, d: 6.797948e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr10 000000000014fe88 (f: 1375880.000000, d: 6.797750e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr12 000003ff8018f918 (f: 2149120256.000000, d: 2.171864e-311)
[2023-05-30T07:46:36.862Z]  [ERR] fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr14 000000000014feb8 (f: 1375928.000000, d: 6.797988e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr15 000000000014fe90 (f: 1375888.000000, d: 6.797790e-318)
[2023-05-30T07:46:36.862Z]  [ERR] Module=/lib64/libpthread.so.0
[2023-05-30T07:46:36.862Z]  [ERR] Module_base_address=000003FFB0A80000 Symbol=raise
[2023-05-30T07:46:36.862Z]  [ERR] Symbol_address=000003FFB0A90B90
[2023-05-30T07:46:36.862Z]  [ERR] Target=2_90_20230529_567 (Linux 3.10.0-1160.90.1.el7.s390x)
[2023-05-30T07:46:36.862Z]  [ERR] CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
[2023-05-30T07:46:36.862Z]  [ERR] ----------- Stack Backtrace -----------
[2023-05-30T07:46:36.862Z]  [ERR] raise+0x30 (0x000003FFB0A90BC0 [libpthread.so.0+0x10bc0])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR4trapEv+0x5e (0x000003FFA8DC2386 [libj9jit29.so+0x6c2386])
[2023-05-30T07:46:36.862Z]  [ERR]  (0x000003FFA8DC23E4 [libj9jit29.so+0x6c23e4])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x402 (0x000003FFA89A95EA [libj9jit29.so+0x2a95ea])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x390 (0x000003FFA89A9A78 [libj9jit29.so+0x2a9a78])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x198e (0x000003FFA8900496 [libj9jit29.so+0x200496])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FFA8883776 [libj9jit29.so+0x183776])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FFA8883C66 [libj9jit29.so+0x183c66])
[2023-05-30T07:46:36.862Z]  [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FFA8883D4A [libj9jit29.so+0x183d4a])
[2023-05-30T07:46:36.862Z]  [ERR] omrsig_protect+0x366 (0x000003FFB0134546 [libj9prt29.so+0x34546])
[2023-05-30T07:46:36.862Z]  [ERR] _Z21compilationThreadProcPv+0x1fe (0x000003FFA88841E6 [libj9jit29.so+0x1841e6])
[2023-05-30T07:46:36.862Z]  [ERR] thread_wrapper+0x114 (0x000003FFB0085A14 [libj9thr29.so+0x5a14])
[2023-05-30T07:46:36.862Z]  [ERR] start_thread+0xea (0x000003FFB0A88312 [libpthread.so.0+0x8312])
[2023-05-30T07:46:36.862Z]  [ERR]  (0x000003FFB098E232 [libc.so.6+0x10e232])
[2023-05-30T07:46:36.862Z]  [ERR] ---------------------------------------
[2023-05-30T07:46:36.862Z]  [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/05/30 00:46:25 - please wait.

[2023-05-30T07:46:43.805Z] ---TEST RESULTS---
[2023-05-30T07:46:43.805Z] Number of PASSED tests: 4 out of 5
[2023-05-30T07:46:43.805Z] Number of FAILED tests: 1 out of 5
[2023-05-30T07:46:43.805Z] 
[2023-05-30T07:46:43.805Z] ---SUMMARY OF FAILED TESTS---
[2023-05-30T07:46:43.805Z] Test -Xnoaot
[2023-05-30T07:46:43.805Z] -----------------------------
[2023-05-30T07:46:43.805Z] 
[2023-05-30T07:46:43.805Z] -----------------------------------
[2023-05-30T07:46:43.805Z] cmdLineTester_criu_jitserverPostRestore_2_FAILED

50x internal grinder - all passed

FYI @tajila

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/345 cmdLineTester_criu_jitserverPostRestore_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/345/functional_test_output.tar.gz

Testing: Test -Xnoaot
Test start time: 2023/06/20 13:49:25 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
Time spent starting: 4 milliseconds
Time spent executing: 31422 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41145
 [OUT] 1874145 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Tue Jun 20 13:49:28 UTC 2023, System.currentTimeMillis(): 1687268968582, System.nanoTime(): 1687268968575296781
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] Post-checkpoint
 [OUT] Removed test output files
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41145
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91: 1874224 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK17_s390x_linux_OMR/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR]  subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xab141e) [0x3ff8ad3141e]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xabf158) [0x3ff8ad3f158]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f71b0) [0x3ff8a9771b0]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f8a8e) [0x3ff8a978a8e]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b35d8) [0x3ff8a5335d8]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3ab6) [0x3ff8a533ab6]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2093b0) [0x3ff8a4893b0]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18dcae) [0x3ff8a40dcae]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e1b4) [0x3ff8a40e1b4]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e24c) [0x3ff8a40e24c]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ea0) [0x3ff8c032ea0]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e68e) [0x3ff8a40e68e]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ff8bf88a5e]
 [ERR] #13: /lib/s390x-linux-gnu/libpthread.so.0(+0x7e66) [0x3ff8ca87e66]
 [ERR] #14: /lib/s390x-linux-gnu/libc.so.6(+0xfcbe6) [0x3ff8c97cbe6]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF8C149400 Handler2=000003FF8C031D48
 [ERR] gpr0=0000000000000005 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF70CFBE90
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF70CFC0D8 gpr7=00000000000019A3
 [ERR] gpr8=000003FF5C016E60 gpr9=000003FF6BE94AC0 gpr10=0000000000000002 gpr11=000003FF70CFBE90
 [ERR] gpr12=000003FF8B15E000 gpr13=000003FF8AEDE578 gpr14=000003FF8A978A3A gpr15=000003FF70CFBDF0
 [ERR] psw=000003FF8CA93B16 mask=0705000180000000 fpc=00080000 bea=000003FF8C9BEE0E
 [ERR] fpr0 000003ffc4cf9670 (f: 3301938688.000000, d: 2.172433e-311)
 [ERR] fpr1 000003ff8afa632c (f: 2331665152.000000, d: 2.171954e-311)
 [ERR] fpr2 000003ffc4cfb81f (f: 3301947392.000000, d: 2.172433e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff70cf8b80 (f: 1892649856.000000, d: 2.171737e-311)
 [ERR] fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr6 000003ff70cf9268 (f: 1892651648.000000, d: 2.171737e-311)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003ff6be80010 (f: 1810366464.000000, d: 2.171696e-311)
 [ERR] fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr10 000003ff70bfd000 (f: 1891618816.000000, d: 2.171736e-311)
 [ERR] fpr11 000003fff2378500 (f: 4063724800.000000, d: 2.172809e-311)
 [ERR] fpr12 000002aa4019f670 (f: 1075443328.000000, d: 1.447732e-311)
 [ERR] fpr13 000003ffa01c2218 (f: 2686198272.000000, d: 2.172129e-311)
 [ERR] fpr14 0000000000000440 (f: 1088.000000, d: 5.375434e-321)
 [ERR] fpr15 000003ffa04d9908 (f: 2689440000.000000, d: 2.172130e-311)
 [ERR] Module=/lib/s390x-linux-gnu/libpthread.so.0
 [ERR] Module_base_address=000003FF8CA80000 Symbol=raise
 [ERR] Symbol_address=000003FF8CA93A90
 [ERR] Target=2_90_20230620_369 (Linux 5.4.0-137-generic)
 [ERR] CPU=s390x (4 logical CPUs) (0x1f58f7000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x86 (0x000003FF8CA93B16 [libpthread.so.0+0x13b16])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF8A978A3A [libj9jit29.so+0x6f8a3a])
 [ERR]  (0x000003FF8A978A94 [libj9jit29.so+0x6f8a94])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF8A5335D8 [libj9jit29.so+0x2b35d8])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x366 (0x000003FF8A533AB6 [libj9jit29.so+0x2b3ab6])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1450 (0x000003FF8A4893B0 [libj9jit29.so+0x2093b0])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF8A40DCAE [libj9jit29.so+0x18dcae])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF8A40E1B4 [libj9jit29.so+0x18e1b4])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF8A40E24C [libj9jit29.so+0x18e24c])
 [ERR] omrsig_protect+0x3e8 (0x000003FF8C032EA0 [libj9prt29.so+0x32ea0])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF8A40E68E [libj9jit29.so+0x18e68e])
 [ERR] thread_wrapper+0xf6 (0x000003FF8BF88A5E [libj9thr29.so+0x8a5e])
 [ERR] start_thread+0xd6 (0x000003FF8CA87E66 [libpthread.so.0+0x7e66])
 [ERR]  (0x000003FF8C97CBE6 [libc.so.6+0xfcbe6])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])
 [ERR] ---------------------------------------
tajila commented 1 year ago

@dsouzai ^^

dsouzai commented 1 year ago

Because of this message:

[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER NO LONGER EXISTS

It looks like the crash is on the JITserver instance, and not on the restored JVM (a crash in the restored JVM wouldn't bring down the jitserver instance).

@mpirvu do you have any ideas as to what could be going on here?

mpirvu commented 1 year ago

I have seen that message when the client was not generating a new clientUID after restore. From the diagnostic data I cannot tell the UID of the client before the checkpoint and after the restore to determine that they are different.

dsouzai commented 1 year ago

I have seen that message when the client was not generating a new clientUID after restore.

Why would that cause the JITServer instance to go down? Shouldn't it just reject the client?

mpirvu commented 1 year ago

The server thinks it's the same client and that it encountered a bug. Rather than reject the client and hide the bug, we prefer to fatal assert.

dsouzai commented 1 year ago

Because this message

2023-05-30T07:46:36.862Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is ready to accept incoming requests

is coming from the jitserver, if it did fatal assert because the client failed to generate a new UID, we should see that assert in the output. However, maybe what I'll do is update the tests to have better separation of the the various things that get cat'd so that we can tell where messages come from.

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/509 cmdLineTester_criu_jitserverPostRestore_2 -Xjit:vlog=vlog

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/509/functional_test_output.tar.gz

TESTING:
*** Starting test suite: J9 Criu Command-Line Post Restore JITServer Option Tests ***
Testing: Generate Verbose Log
Test start time: 2023/06/28 01:14:41 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=vlog" 1 false true
Time spent starting: 76 milliseconds
Time spent executing: 35743 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=50096
 [OUT] 1247254 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Jun 28 01:14:45 UTC 2023, System.currentTimeMillis(): 1687914885074, System.nanoTime(): 1687914885071522333
 [OUT] Post-checkpoint
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=50096
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91: 1247333 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK17_s390x_linux_Nightly/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR]  subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xab141e) [0x3ff92ab141e]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xabf158) [0x3ff92abf158]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f7260) [0x3ff926f7260]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f8b3e) [0x3ff926f8b3e]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3608) [0x3ff922b3608]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3ae6) [0x3ff922b3ae6]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2093e0) [0x3ff922093e0]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18dcae) [0x3ff9218dcae]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e1b4) [0x3ff9218e1b4]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e24c) [0x3ff9218e24c]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ea0) [0x3ff93db2ea0]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e68e) [0x3ff9218e68e]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ff93d08a5e]
 [ERR] #13: /lib/s390x-linux-gnu/libpthread.so.0(+0x7e66) [0x3ff94807e66]
 [ERR] #14: /lib/s390x-linux-gnu/libc.so.6(+0xfcbe6) [0x3ff946fcbe6]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF93EC9400 Handler2=000003FF93DB1D48
 [ERR] gpr0=0000000000000005 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF78A7BE90
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF78A7C0D8 gpr7=000000000000196F
 [ERR] gpr8=000003FF6401FCE8 gpr9=000003FEFA297B80 gpr10=0000000000000002 gpr11=000003FF78A7BE90
 [ERR] gpr12=000003FF92EDE000 gpr13=000003FF92C5E778 gpr14=000003FF926F8AEA gpr15=000003FF78A7BDF0
 [ERR] psw=000003FF94813B16 mask=0705000180000000 fpc=00080000 bea=000003FF9473EE0E
 [ERR] fpr0 000003ffec3fa240 (f: 3963593216.000000, d: 2.172760e-311)
 [ERR] fpr1 000003ff92d26828 (f: 2463262720.000000, d: 2.172019e-311)
 [ERR] fpr2 000003ffec3fc569 (f: 3963602176.000000, d: 2.172760e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff78a78b80 (f: 2024246144.000000, d: 2.171802e-311)
 [ERR] fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr6 000003ff78a79268 (f: 2024247936.000000, d: 2.171802e-311)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003fefa280010 (f: 4196925440.000000, d: 2.170753e-311)
 [ERR] fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr10 000003ff7897d000 (f: 2023215104.000000, d: 2.171801e-311)
 [ERR] fpr11 000003ffeb7f8ee0 (f: 3951005440.000000, d: 2.172754e-311)
 [ERR] fpr12 000002aa0ee44960 (f: 249842016.000000, d: 1.447325e-311)
 [ERR] fpr13 000003ff984d56d8 (f: 2555205376.000000, d: 2.172064e-311)
 [ERR] fpr14 0000000000000440 (f: 1088.000000, d: 5.375434e-321)
 [ERR] fpr15 000003ff98780928 (f: 2558003456.000000, d: 2.172066e-311)
 [ERR] Module=/lib/s390x-linux-gnu/libpthread.so.0
 [ERR] Module_base_address=000003FF94800000 Symbol=raise
 [ERR] Symbol_address=000003FF94813A90
 [ERR] Target=2_90_20230627_518 (Linux 5.4.0-137-generic)
 [ERR] CPU=s390x (4 logical CPUs) (0x1f58f7000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x86 (0x000003FF94813B16 [libpthread.so.0+0x13b16])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF926F8AEA [libj9jit29.so+0x6f8aea])
 [ERR]  (0x000003FF926F8B44 [libj9jit29.so+0x6f8b44])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF922B3608 [libj9jit29.so+0x2b3608])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x366 (0x000003FF922B3AE6 [libj9jit29.so+0x2b3ae6])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1450 (0x000003FF922093E0 [libj9jit29.so+0x2093e0])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF9218DCAE [libj9jit29.so+0x18dcae])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF9218E1B4 [libj9jit29.so+0x18e1b4])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF9218E24C [libj9jit29.so+0x18e24c])
 [ERR] omrsig_protect+0x3e8 (0x000003FF93DB2EA0 [libj9prt29.so+0x32ea0])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF9218E68E [libj9jit29.so+0x18e68e])
 [ERR] thread_wrapper+0xf6 (0x000003FF93D08A5E [libj9thr29.so+0x8a5e])
 [ERR] start_thread+0xd6 (0x000003FF94807E66 [libpthread.so.0+0x7e66])
 [ERR]  (0x000003FF946FCBE6 [libc.so.6+0xfcbe6])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])
JasonFengJ9 commented 1 year ago

JDK21 s390x_linux(rhel7s390x-svl-rt4-1)

variation: -Xjit
JVM_OPTIONS:  -Xjit 

TESTING:
*** Starting test suite: J9 Criu Command-Line Post Restore JITServer Option Tests ***
Testing: Generate Verbose Log
Test start time: 2023/09/03 06:24:50 Pacific Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=vlog" 1 false true
Time spent starting: 113 milliseconds
Time spent executing: 10294 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=45559
 [OUT]  6393 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Sun Sep 03 06:24:53 PDT 2023, System.currentTimeMillis(): 1693747493699, System.nanoTime(): 1693747378908550128
 [OUT] Post-checkpoint
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=45559
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91:  6472 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK21_s390x_linux_Personal/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR]  subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaba7ae) [0x3ff8363a7ae]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xac85a0) [0x3ff836485a0]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f78a4) [0x3ff832778a4]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f9196) [0x3ff83279196]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b06b8) [0x3ff82e306b8]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b0bb4) [0x3ff82e30bb4]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x20b7ec) [0x3ff82d8b7ec]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18fbc6) [0x3ff82d0fbc6]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1900cc) [0x3ff82d100cc]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x190164) [0x3ff82d10164]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b28) [0x3ff84832b28]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1905a6) [0x3ff82d105a6]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8efe) [0x3ff84788efe]
 [ERR] #13: /lib64/libpthread.so.0(+0x9ad8) [0x3ff85289ad8]
 [ERR] #14: /lib64/libc.so.6(+0x10baee) [0x3ff8518baee]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF84949878 Handler2=000003FF848319D8
 [ERR] gpr0=000003FF689FBE50 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF689FBE50
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF689FC098 gpr7=000000000000367B
 [ERR] gpr8=000003FF54010948 gpr9=000003FF63E8CD40 gpr10=0000000000000002 gpr11=000003FF83A86020
 [ERR] gpr12=000003FF83A4AF48 gpr13=000003FF852991A0 gpr14=000003FF83279142 gpr15=000003FF689FBDB0
 [ERR] psw=000003FF85295E5E mask=0705100180000000 fpc=00080000 bea=000003FF850FABC2
 [ERR] fpr0 0000000000000005 (f: 5.000000, d: 2.470328e-323)
 [ERR] fpr1 000003ff54000e08 (f: 1409289728.000000, d: 2.171498e-311)
 [ERR] fpr2 000003fff9f7dc5a (f: 4193770496.000000, d: 2.172874e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff689f8b10 (f: 1755286272.000000, d: 2.171669e-311)
 [ERR] fpr5 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
 [ERR] fpr6 0000000000000007 (f: 7.000000, d: 3.458460e-323)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003ff63e80010 (f: 1676148736.000000, d: 2.171630e-311)
 [ERR] fpr9 000003ffc2b77978 (f: 3266803968.000000, d: 2.172416e-311)
 [ERR] fpr10 000003ff688fd000 (f: 1754255360.000000, d: 2.171668e-311)
 [ERR] fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr12 000002aa2ccb3470 (f: 751514752.000000, d: 1.447572e-311)
 [ERR] fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr14 000002aa2ccaeea0 (f: 751496832.000000, d: 1.447572e-311)
 [ERR] fpr15 000003ff8c7cba28 (f: 2356984320.000000, d: 2.171966e-311)
 [ERR] Module=/lib64/libpthread.so.0
 [ERR] Module_base_address=000003FF85280000 Symbol=raise
 [ERR] Symbol_address=000003FF85295DD0
 [ERR] Target=2_90_20230903_61 (Linux 5.14.21-150400.24.69-default)
 [ERR] CPU=s390x (4 logical CPUs) (0x1eaf17000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x8e (0x000003FF85295E5E [libpthread.so.0+0x15e5e])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF83279142 [libj9jit29.so+0x6f9142])
 [ERR]  (0x000003FF8327919C [libj9jit29.so+0x6f919c])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF82E306B8 [libj9jit29.so+0x2b06b8])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x384 (0x000003FF82E30BB4 [libj9jit29.so+0x2b0bb4])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x13cc (0x000003FF82D8B7EC [libj9jit29.so+0x20b7ec])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF82D0FBC6 [libj9jit29.so+0x18fbc6])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF82D100CC [libj9jit29.so+0x1900cc])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF82D10164 [libj9jit29.so+0x190164])
 [ERR] omrsig_protect+0x3d8 (0x000003FF84832B28 [libj9prt29.so+0x32b28])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF82D105A6 [libj9jit29.so+0x1905a6])
 [ERR] thread_wrapper+0xf6 (0x000003FF84788EFE [libj9thr29.so+0x8efe])
 [ERR] start_thread+0xd8 (0x000003FF85289AD8 [libpthread.so.0+0x9ad8])
 [ERR]  (0x000003FF8518BAEE [libc.so.6+0x10baee])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])
 [ERR] ---------------------------------------
 [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 06:24:56 - please wait.

>> Success condition was found: [Output match: Killed]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was found: [Output match: Post-checkpoint]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Success condition was found: [Output match: JITSERVER EXISTS]
>> Success condition was not found: [Output match: JITSERVER STILL EXISTS]
>> Failure condition was not found: [Output match: JITSERVER DOES NOT EXIST]
>> Failure condition was found: [Output match: JITSERVER NO LONGER EXISTS]
>> Failure condition was not found: [Output match: User requested Java dump using]

---TEST RESULTS---
Number of PASSED tests: 4 out of 5
Number of FAILED tests: 1 out of 5

---SUMMARY OF FAILED TESTS---
Generate Verbose Log
-----------------------------

-----------------------------------
cmdLineTester_criu_jitserverPostRestore_0_FAILED
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_aarch64_linux_Nightly_testList_0/93

pshipton commented 11 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/678

pshipton commented 10 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_aarch64_linux_Release/83

JasonFengJ9 commented 10 months ago

From an internal build(ubu22s390x-svl-rt8-1):

java version "21.0.1" 2023-10-17 LTS
IBM Semeru Runtime Certified Edition 21.0.1.0 (build 21.0.1+12-LTS)
Eclipse OpenJ9 VM 21.0.1.0 (build openj9-0.42.0, JRE 21 Linux s390x-64-Bit Compressed References 20231017_77 (JIT enabled, AOT enabled)
OpenJ9   - 874af12ee
OMR      - 11700e64f
JCL      - eb47ed546b0 based on jdk-21.0.1+12)

[2024-01-09T23:16:52.719Z]  [ERR] JITServer is ready to accept incoming requests
[2024-01-09T23:16:52.719Z]  [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 201175 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
[2024-01-09T23:16:52.719Z]  [ERR] Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk21u/jdk21u-linux-s390x-openj9-IBM/workspace/build/src/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
[2024-01-09T23:16:52.719Z]  [ERR]   subclass info cannot be null: ensure subclasses are loaded before superclass
tajila commented 10 months ago

FYI @dsouzai 

subclass info cannot be null: ensure subclasses are loaded before superclass
pshipton commented 8 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/54 - ub18-x86-2 cmdLineTester_criu_jitserverPostRestore_3

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/54/functional_test_output.tar.gz

Testing: Test -Xnoaot
Test start time: 2024/03/06 09:43:13 Atlantic Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/bin " -XX:+JVMPortableRestoreMode " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
Time spent starting: 3 milliseconds
Time spent executing: 15808 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=48769 
 [OUT] 15720 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] main: Wed Mar 06 09:43:16 AST 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1709732596554, System.nanoTime(): 26673603211900625
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] Post-checkpoint
 [OUT] Removed test output files
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=48769 
 [OUT] finished script
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 113: 15798 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK21_x86-64_linux_OMR/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR]  subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x928fc5) [0x7f3dd55aefc5]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x935370) [0x7f3dd55bb370]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x5dd86a) [0x7f3dd526386a]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x5dee0d) [0x7f3dd5264e0d]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x27a671) [0x7f3dd4f00671]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x27aa33) [0x7f3dd4f00a33]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e7706) [0x7f3dd4e6d706]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17621b) [0x7f3dd4dfc21b]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176582) [0x7f3dd4dfc582]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176632) [0x7f3dd4dfc632]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x2a3b9) [0x7f3dd7cb93b9]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1769fb) [0x7f3dd4dfc9fb]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_x86-64_linux_OMR_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0xb3a3) [0x7f3dd7c813a3]
 [ERR] #13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3dd773a6db]
 [ERR] #14: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3dd746361f]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=00007F3DD67D37C0 Handler2=00007F3DD7CB8730
 [ERR] RDI=0000000000000002 RSI=00007F3DBAA0E2C0 RAX=0000000000000000 RBX=00007F3D503B2420
 [ERR] RCX=00007F3DD7745817 RDX=0000000000000000 R8=0000000000000000 R9=00007F3DBAA0E2C0
 [ERR] R10=0000000000000008 R11=0000000000000246 R12=0000000000000007 R13=00007F3D503CC140
 [ERR] R14=00007F3DB40128E0 R15=00007F3DB4029A20
 [ERR] RIP=00007F3DD7745817 GS=0000 FS=0000 RSP=00007F3DBAA0E2C0
 [ERR] EFlags=0000000000000246 CS=0033 RBP=00007F3DD4EFEE00 ERR=0000000000000000
 [ERR] TRAPNO=0000000000000000 OLDMASK=0000000000000000 CR2=0000000000000000
 [ERR] xmm0 ffffffffffffffff (f: 4294967296.000000, d: -nan)
 [ERR] xmm1 00000000000000ff (f: 255.000000, d: 1.259867e-321)
 [ERR] xmm2 67756265446f4e5f (f: 1148145280.000000, d: 2.381955e+190)
 [ERR] xmm3 ffff000000000000 (f: 0.000000, d: -nan)
 [ERR] xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] xmm5 2f65636170736b72 (f: 1886612352.000000, d: 2.254787e-80)
 [ERR] xmm6 6a5f31326b646a6e (f: 1801742976.000000, d: 2.444909e+204)
 [ERR] xmm7 6f6974636e75662e (f: 1853187584.000000, d: 4.824071e+228)
 [ERR] xmm8 646433663778305b (f: 930623552.000000, d: 3.997013e+175)
 [ERR] xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] xmm10 00007f3db40295c0 (f: 3020068352.000000, d: 6.912126e-310)
 [ERR] xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] xmm13 000001380000013f (f: 319.000000, d: 6.620627e-312)
 [ERR] xmm14 0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
 [ERR] xmm15 000001420000013b (f: 315.000000, d: 6.832826e-312)
 [ERR] Module=/lib/x86_64-linux-gnu/libpthread.so.0
 [ERR] Module_base_address=00007F3DD7733000 Symbol=raise
 [ERR] Symbol_address=00007F3DD7745750
 [ERR] Target=2_90_20240306_65 (Linux 4.15.0-173-generic)
 [ERR] CPU=amd64 (4 logical CPUs) (0x5e27ee000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0xc7 (0x00007F3DD7745817 [libpthread.so.0+0x12817])
 [ERR] _ZN2TR4trapEv+0x47 (0x00007F3DD5264D87 [libj9jit29.so+0x5ded87])
 [ERR]  (0x00007F3DD5264E12 [libj9jit29.so+0x5dee12])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2b1 (0x00007F3DD4F00671 [libj9jit29.so+0x27a671])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x2a3 (0x00007F3DD4F00A33 [libj9jit29.so+0x27aa33])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xd86 (0x00007F3DD4E6D706 [libj9jit29.so+0x1e7706])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x35b (0x00007F3DD4DFC21B [libj9jit29.so+0x17621b])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x00007F3DD4DFC582 [libj9jit29.so+0x176582])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x00007F3DD4DFC632 [libj9jit29.so+0x176632])
 [ERR] omrsig_protect+0x239 (0x00007F3DD7CB93B9 [libj9prt29.so+0x2a3b9])
 [ERR] _Z21compilationThreadProcPv+0x17b (0x00007F3DD4DFC9FB [libj9jit29.so+0x1769fb])
 [ERR] thread_wrapper+0x163 (0x00007F3DD7C813A3 [libj9thr29.so+0xb3a3])
 [ERR] start_thread+0xdb (0x00007F3DD773A6DB [libpthread.so.0+0x76db])
 [ERR] clone+0x3f (0x00007F3DD746361F [libc.so.6+0x12161f])
 [ERR] ---------------------------------------
pshipton commented 8 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_1/736 cmdLineTester_criu_jitserverPostRestore_3

dsouzai commented 8 months ago

Reopening since I'm not sure if #19088 does actually fix it or not; it's a speculative PR. I guess this got linked and was closed automatically.

pshipton commented 8 months ago

https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests doesn't understand "may fix".

pshipton commented 8 months ago

Another assertion https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/621

pshipton commented 8 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/4 cmdLineTester_criu_jitserverPostRestore_2

Testing: Test -Xnoaot
Test start time: 2024/03/17 15:11:30 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/jdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
...
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 10893 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK22_s390x_linux_Release/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR]  subclass info cannot be null: ensure subclasses are loaded before superclass
JasonFengJ9 commented 8 months ago

JDK17 s390x_linux(ubu22s390x-svl-rt8-1)

[2024-04-02T01:12:49.553Z] variation: -Xjit:vlog=vlog
[2024-04-02T01:12:49.553Z] JVM_OPTIONS:  -Xjit:vlog=vlog 

[2024-04-02T01:13:04.870Z] Testing: Test -Xnoaot
[2024-04-02T01:13:04.870Z] Test start time: 2024/04/02 01:13:03 Coordinated Universal Time
[2024-04-02T01:13:04.870Z] Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
[2024-04-02T01:13:04.870Z] Time spent starting: 3 milliseconds
[2024-04-02T01:13:14.819Z] Time spent executing: 10596 milliseconds
[2024-04-02T01:13:14.819Z] Test result: FAILED
[2024-04-02T01:13:14.819Z] Output from test:
[2024-04-02T01:13:14.819Z]  [OUT] start running script
[2024-04-02T01:13:14.819Z]  [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
[2024-04-02T01:13:14.819Z]  [OUT] export LD_BIND_NOT=on
[2024-04-02T01:13:14.819Z]  [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=44959 -XX:JITServerHealthProbePort=47076 
[2024-04-02T01:13:14.819Z]  [OUT] 3735070 ?        00:00:00 jitserver
[2024-04-02T01:13:14.819Z]  [OUT] JITSERVER EXISTS
[2024-04-02T01:13:14.819Z]  [OUT] Pre-checkpoint
[2024-04-02T01:13:14.819Z]  [OUT] main: Tue Apr 02 01:13:06 UTC 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1712020386429, System.nanoTime(): 1712020275214754523
[2024-04-02T01:13:14.819Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2024-04-02T01:13:14.819Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2024-04-02T01:13:14.819Z]  [OUT] Post-checkpoint
[2024-04-02T01:13:14.819Z]  [OUT] Removed test output files
[2024-04-02T01:13:14.819Z]  [OUT] JITSERVER NO LONGER EXISTS
[2024-04-02T01:13:14.819Z]  [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=44959 -XX:JITServerHealthProbePort=47076 
[2024-04-02T01:13:14.819Z]  [OUT] finished script
[2024-04-02T01:13:14.819Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2024-04-02T01:13:14.819Z]  [ERR] 
[2024-04-02T01:13:14.819Z]  [ERR] JITServer is ready to accept incoming requests
[2024-04-02T01:13:14.819Z]  [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 3735148 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
[2024-04-02T01:13:14.819Z]  [ERR] Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk17u/jdk17u-linux-s390x-openj9-IBM/workspace/build/src/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
[2024-04-02T01:13:14.819Z]  [ERR]   subclass info cannot be null: ensure subclasses are loaded before superclass

[2024-04-02T01:15:37.551Z] ---TEST RESULTS---
[2024-04-02T01:15:37.551Z] Number of PASSED tests: 10 out of 11
[2024-04-02T01:15:37.551Z] Number of FAILED tests: 1 out of 11
[2024-04-02T01:15:37.551Z] 
[2024-04-02T01:15:37.551Z] ---SUMMARY OF FAILED TESTS---
[2024-04-02T01:15:37.551Z] Test -Xnoaot
[2024-04-02T01:15:37.551Z] -----------------------------
[2024-04-02T01:15:37.551Z] 
[2024-04-02T01:15:37.551Z] -----------------------------------
[2024-04-02T01:15:37.551Z] cmdLineTester_criu_jitserverPostRestore_2_FAILED
pshipton commented 7 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_sanity.functional_s390x_linux_Nightly_testList_0/67

pshipton commented 7 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_aarch64_linux_Release_testList_0/11 This is a 0.45 release build.

pshipton commented 6 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_1/779

pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_1/800

vij-singh commented 2 months ago

@dsouzai Is this something we're looking at for the 0.48 release?

dsouzai commented 2 months ago

No; I spent some time a few months ago looking at this but couldn't make much progress. It's fairly intermittent and seems to only happen with JITServer + CRIU.

pshipton commented 1 month ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/122 cmdLineTester_criu_jitserverPostRestore_3

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/122/functional_test_output.tar.gz

01:08:55  Testing: Test -Xnoaot
01:08:55  Test start time: 2024/09/29 01:08:55 Eastern Standard Time
01:08:55  Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin " -XX:+JVMPortableRestoreMode " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
01:08:55  Time spent starting: 2 milliseconds
01:09:11  Time spent executing: 14139 milliseconds
01:09:11  Test result: FAILED
01:09:11  Output from test:
01:09:11   [OUT] start running script
01:09:11   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
01:09:11   [OUT] export LD_BIND_NOT=on
01:09:11   [OUT] Starting /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41861 -XX:JITServerHealthProbePort=40028 
01:09:11   [OUT] 1608135 ?        00:00:00 jitserver
01:09:11   [OUT] JITSERVER EXISTS
01:09:11   [OUT] Pre-checkpoint
01:09:11   [OUT] main: Sun Sep 29 01:08:57 EDT 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1727586537385, System.nanoTime(): 1727586537382660007
01:09:11   [OUT] JVMJITM043W AOT load and compilation disabled post restore.
01:09:11   [OUT] Post-checkpoint
01:09:11   [OUT] Removed test output files
01:09:11   [OUT] JITSERVER NO LONGER EXISTS
01:09:11   [OUT] Terminating /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41861 -XX:JITServerHealthProbePort=40028 
01:09:11   [OUT] finished script
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
01:09:11   [ERR] JITServer is currently a technology preview. Its use is not yet supported.
01:09:11   [ERR] 
01:09:11   [ERR] JITServer is ready to accept incoming requests
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 1608215 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
01:09:11   [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK11_s390x_linux_Release/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
01:09:11   [ERR]    subclass info cannot be null: ensure subclasses are loaded before superclass
01:09:11   [ERR] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xada82e) [0x3ffa345a82e]
01:09:11   [ERR] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xae87d0) [0x3ffa34687d0]
01:09:11   [ERR] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x710dcc) [0x3ffa3090dcc]
01:09:11   [ERR] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x7126be) [0x3ffa30926be]
01:09:11   [ERR] #4: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2bd470) [0x3ffa2c3d470]
01:09:11   [ERR] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2bd96c) [0x3ffa2c3d96c]
01:09:11   [ERR] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x210c92) [0x3ffa2b90c92]
01:09:11   [ERR] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193432) [0x3ffa2b13432]
01:09:11   [ERR] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193924) [0x3ffa2b13924]
01:09:11   [ERR] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1939bc) [0x3ffa2b139bc]
01:09:11   [ERR] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32bb0) [0x3ffa46b2bb0]
01:09:11   [ERR] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193dfe) [0x3ffa2b13dfe]
01:09:11   [ERR] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a46) [0x3ffa4608a46]
01:09:11   [ERR] #13: /lib64/libpthread.so.0(+0x80de) [0x3ffa50880de]
01:09:11   [ERR] #14: /lib64/libc.so.6(+0x28b42) [0x3ffa4ea8b42]
01:09:11   [ERR] #15: [(nil)]
01:09:11   [ERR] 
01:09:11   [ERR] Unhandled exception
01:09:11   [ERR] Type=Unhandled trap vmState=0x00000000
01:09:11   [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
01:09:11   [ERR] Handler1=000003FFA47C9618 Handler2=000003FFA46B1A60
01:09:11   [ERR] gpr0=000003FF8977DE48 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF8977DE48
01:09:11   [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF8977E090 gpr7=000003FF84012060
01:09:11   [ERR] gpr8=000003FF84019480 gpr9=000003FF0A494900 gpr10=0000000000000002 gpr11=000003FFA38C1010
01:09:11   [ERR] gpr12=000003FFA3885000 gpr13=000003FFA50961F8 gpr14=000003FFA309266A gpr15=000003FF8977DDA8
01:09:11   [ERR] psw=000003FFA5093244 mask=0705000180000000 fpc=00080000 bea=000003FFA4FC6AF8
01:09:11   [ERR] fpr0=0000000000000005 (f: 5.000000, d: 2.470328e-323)
01:09:11   [ERR] fpr1=0000000000000001 (f: 1.000000, d: 4.940656e-324)
01:09:11   [ERR] fpr2=000003ffa501eca8 (f: 2768366848.000000, d: 2.172169e-311)
01:09:11   [ERR] fpr3=0000000000000007 (f: 7.000000, d: 3.458460e-323)
01:09:11   [ERR] fpr4=000003ff8977b220 (f: 2306322944.000000, d: 2.171941e-311)
01:09:11   [ERR] fpr5=000003ff84000de8 (f: 2214596096.000000, d: 2.171896e-311)
01:09:11   [ERR] fpr6=000003ff00000001 (f: 1.000000, d: 2.170802e-311)
01:09:11   [ERR] fpr7=d1d6b8188c58eefd (f: 2354638592.000000, d: -1.765429e+86)
01:09:11   [ERR] fpr8=000003ff0a480010 (f: 172490768.000000, d: 2.170887e-311)
01:09:11   [ERR] fpr9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:09:11   [ERR] fpr10=00000000a0e15260 (f: 2699121152.000000, d: 1.333543e-314)
01:09:11   [ERR] fpr11=000003ff8b7f1f28 (f: 2340364032.000000, d: 2.171958e-311)
01:09:11   [ERR] fpr12=000003ff8977f910 (f: 2306341120.000000, d: 2.171941e-311)
01:09:11   [ERR] fpr13=0000000000000001 (f: 1.000000, d: 4.940656e-324)
01:09:11   [ERR] fpr14=0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
01:09:11   [ERR] fpr15=000003ff84590968 (f: 2220427520.000000, d: 2.171899e-311)
01:09:11   [ERR] Module=/lib64/libpthread.so.0
01:09:11   [ERR] Module_base_address=000003FFA5080000 Symbol=raise
01:09:11   [ERR] Symbol_address=000003FFA50931B8
01:09:11   [ERR] Target=2_90_20240928_144 (Linux 4.18.0-513.24.1.el8_9.s390x)
01:09:11   [ERR] CPU=s390x (4 logical CPUs) (0x1d5ba6000 RAM)
01:09:11   [ERR] ----------- Stack Backtrace -----------
01:09:11   [ERR] raise+0x8c (0x000003FFA5093244 [libpthread.so.0+0x13244])
01:09:11   [ERR] _ZN2TR4trapEv+0x52 (0x000003FFA309266A [libj9jit29.so+0x71266a])
01:09:11   [ERR]  (0x000003FFA30926C4 [libj9jit29.so+0x7126c4])
01:09:11   [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FFA2C3D470 [libj9jit29.so+0x2bd470])
01:09:11   [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x384 (0x000003FFA2C3D96C [libj9jit29.so+0x2bd96c])
01:09:11   [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xd82 (0x000003FFA2B90C92 [libj9jit29.so+0x210c92])
01:09:11   [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x402 (0x000003FFA2B13432 [libj9jit29.so+0x193432])
01:09:11   [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FFA2B13924 [libj9jit29.so+0x193924])
01:09:11   [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FFA2B139BC [libj9jit29.so+0x1939bc])
01:09:11   [ERR] omrsig_protect+0x3d8 (0x000003FFA46B2BB0 [libj9prt29.so+0x32bb0])
01:09:11   [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FFA2B13DFE [libj9jit29.so+0x193dfe])
01:09:11   [ERR] thread_wrapper+0xf6 (0x000003FFA4608A46 [libj9thr29.so+0x8a46])
01:09:11   [ERR] start_thread+0xd6 (0x000003FFA50880DE [libpthread.so.0+0x80de])
01:09:11   [ERR]  (0x000003FFA4EA8B42 [libc.so.6+0x28b42])
01:09:11   [ERR]  (0x0000000000000000 [<unknown>+0x0])
01:09:11   [ERR] ---------------------------------------
pshipton commented 4 weeks ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/231

hangshao0 commented 2 weeks ago

Another one: https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_sanity.functional_ppc64le_linux_vt_standard_Personal_testList_1/29/consoleFull

pshipton commented 1 week ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_1/37

pshipton commented 2 days ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/41/ cmdLineTester_criu_jitserverAcrossCheckpoint_0 - ubu24-ppc64le-4

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/41/functional_test_output.tar.gz

22:53:53  Testing: Enable JITServer specified Pre-Checkpoint but not explicitly enabled Post-Restore
22:53:53  Test start time: 2024/11/23 03:53:53 Coordinated Universal Time
22:53:53  Running command: bash /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin "  -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={JITServer},verbose={JITServerConns},vlog=preCheckpointVlog" org.openj9.criu.OptionsFileTest "JitOptionsTest -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=postRestoreVlog" 1 false true
22:53:53  Time spent starting: 2 milliseconds
22:54:03  Time spent executing: 9618 milliseconds
22:54:03  Test result: FAILED
22:54:03  Output from test:
22:54:03   [OUT] start running script
22:54:03   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
22:54:03   [OUT] export LD_BIND_NOT=on
22:54:03   [OUT] Starting /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=57426 -XX:JITServerHealthProbePort=39121 
22:54:03   [OUT] 2262857 ?        00:00:00 jitserver
22:54:03   [OUT] JITSERVER EXISTS
22:54:03   [OUT] Pre-checkpoint
22:54:03   [OUT] main: Sat Nov 23 03:53:55 UTC 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1732334035877, System.nanoTime(): 5698589620837107
22:54:03   [OUT] Post-checkpoint
22:54:03   [OUT] JITSERVER NO LONGER EXISTS
22:54:03   [OUT] Terminating /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=57426 -XX:JITServerHealthProbePort=39121 
22:54:03   [OUT] finished script
22:54:03   [ERR] 
22:54:03   [ERR] JITServer is ready to accept incoming requests
22:54:03   [ERR] /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 2262939 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
22:54:03   [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK23_ppc64le_linux_Nightly/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
22:54:03   [ERR]    subclass info cannot be null: ensure subclasses are loaded before superclass
22:54:03   [ERR] #0: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb7cdc0) [0x64b92817cdc0]
22:54:03   [ERR] #1: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb8e80c) [0x64b92818e80c]
22:54:03   [ERR] #2: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x765dfc) [0x64b927d65dfc]
22:54:03   [ERR] #3: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x767aac) [0x64b927d67aac]
22:54:03   [ERR] #4: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2dd9d0) [0x64b9278dd9d0]
22:54:03   [ERR] #5: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2ddefc) [0x64b9278ddefc]
22:54:03   [ERR] #6: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x202568) [0x64b927802568]
22:54:03   [ERR] #7: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x175e80) [0x64b927775e80]
22:54:03   [ERR] #8: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176498) [0x64b927776498]
22:54:03   [ERR] #9: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176550) [0x64b927776550]
22:54:03   [ERR] #10: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x39cd4) [0x64b928d69cd4]
22:54:03   [ERR] #11: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176ab8) [0x64b927776ab8]
22:54:03   [ERR] #12: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0xcc00) [0x64b92959cc00]
22:54:03   [ERR] #13: /lib/powerpc64le-linux-gnu/libc.so.6(+0xb2a5c) [0x64b9296b2a5c]
22:54:03   [ERR] 
22:54:03   [ERR] Unhandled exception
22:54:03   [ERR] Type=Unhandled trap vmState=0x00000000
22:54:03   [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
22:54:03   [ERR] Handler1=000064B928E42C40 Handler2=000064B928D68820
22:54:03   [ERR] R0=00000000000000FA R1=000064B90C1FB510 R2=000064B929876D00 R3=0000000000000000
22:54:03   [ERR] R4=000000000022874D R5=0000000000000005 R6=2D3D544547524154 R7=0000000000000005
22:54:03   [ERR] R8=002D3D5445475241 R9=0000000000000000 R10=0000000000000000 R11=0000000000000000
22:54:03   [ERR] R12=0000000000000000 R13=000064B90C2068E0 R14=0000000000000001 R15=00000000002AE100
22:54:03   [ERR] R16=000064B9285C0778 R17=000064B9285EBAD8 R18=0000000000000001 R19=000064B90D109168
22:54:03   [ERR] R20=0000000000000000 R21=000064B90C1FB640 R22=000064B90C1FB688 R23=000064B90C1FB658
22:54:03   [ERR] R24=0000000000000030 R25=000064B900012E00 R26=000064B9278DB950 R27=000064B90001A7B0
22:54:03   [ERR] R28=0000000000000005 R29=000064B90001A8B8 R30=000064B90C1FF120 R31=000000000022874D
22:54:03   [ERR] NIP=000064B9296B55BC MSR=800000000280D033 ORIG_GPR3=0000000000228749 CTR=0000000000000000
22:54:03   [ERR] LINK=000064B9296B55BC XER=0000000000000000 CCR=0000000048884408 SOFTE=0000000000000001
22:54:03   [ERR] TRAP=0000000000003000 DAR=000064B928DD3C4C dsisr=0000000040000000 RESULT=0000000000000000
22:54:03   [ERR] FPR0=0000000000357e00 (f: 3505664.000000, d: 1.732028e-317)
22:54:03   [ERR] FPR1=3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
22:54:03   [ERR] FPR2=4113400000000000 (f: 0.000000, d: 3.153920e+05)
22:54:03   [ERR] FPR3=0000002e00000021 (f: 33.000000, d: 9.761181e-313)
22:54:03   [ERR] FPR4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR5=0000002600000027 (f: 39.000000, d: 8.063584e-313)
22:54:03   [ERR] FPR6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR11=5b20296335613262 (f: 895562368.000000, d: 8.962195e+130)
22:54:03   [ERR] FPR12=6b646a2f305f7473 (f: 811562112.000000, d: 2.097354e+209)
22:54:03   [ERR] FPR13=616d692d6b647332 (f: 1801745152.000000, d: 2.067451e+161)
22:54:03   [ERR] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] Module=/lib/powerpc64le-linux-gnu/libc.so.6
22:54:03   [ERR] Module_base_address=000064B929600000 Symbol=pthread_kill
22:54:03   [ERR] Symbol_address=000064B9296B5430
22:54:03   [ERR] Target=2_90_20241123_43 (Linux 6.8.0-45-generic)
22:54:03   [ERR] CPU=ppc64le (4 logical CPUs) (0x1ea0e0000 RAM)
22:54:03   [ERR] ----------- Stack Backtrace -----------
22:54:03   [ERR] pthread_kill+0x18c (0x000064B9296B55BC [libc.so.6+0xb55bc])
22:54:03   [ERR] gsignal+0x2c (0x000064B92964BD5C [libc.so.6+0x4bd5c])
22:54:03   [ERR] _ZN2TR4trapEv+0x70 (0x000064B927D67A60 [libj9jit29.so+0x767a60])
22:54:03   [ERR] _ZN2TR15fatal_assertionEPKciS1_S1_z+0x30 (0x000064B927D67AB0 [libj9jit29.so+0x767ab0])
22:54:03   [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2e0 (0x000064B9278DD9D0 [libj9jit29.so+0x2dd9d0])
22:54:03   [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x37c (0x000064B9278DDEFC [libj9jit29.so+0x2ddefc])
22:54:03   [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xa18 (0x000064B927802568 [libj9jit29.so+0x202568])
22:54:03   [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x410 (0x000064B927775E80 [libj9jit29.so+0x175e80])
22:54:03   [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xa8 (0x000064B927776498 [libj9jit29.so+0x176498])
22:54:03   [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0xa0 (0x000064B927776550 [libj9jit29.so+0x176550])
22:54:03   [ERR] omrsig_protect+0x3e4 (0x000064B928D69CD4 [libj9prt29.so+0x39cd4])
22:54:03   [ERR] _Z21compilationThreadProcPv+0x1a8 (0x000064B927776AB8 [libj9jit29.so+0x176ab8])
22:54:03   [ERR] thread_wrapper+0x190 (0x000064B92959CC00 [libj9thr29.so+0xcc00])
22:54:03   [ERR]  (0x000064B9296B2A5C [libc.so.6+0xb2a5c])
22:54:03   [ERR] ---------------------------------------