search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

cmdLineTester_criu_nonPortableRestore MethodTypeDeadlockTest crash #18800

Closed pshipton closed 8 months ago

pshipton commented 9 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/588/ cmdLineTester_criu_nonPortableRestore_8 -Xgcpolicy:gencon -Xgcthreads1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/588/functional_test_output.tar.gz

Testing: Create and Restore Criu Checkpoint Image once - MethodTypeDeadlockTest
Test start time: 2024/01/22 22:06:55 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java " -Xgcpolicy:gencon -Xgcthreads1  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED" org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
Time spent starting: 2 milliseconds
Time spent executing: 23557 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xgcpolicy:gencon -Xgcthreads1  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED -cp /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
 [OUT] Pre-checkpoint
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x00000000
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF94D11E44 Handler2=0000FFFF94C77B60 InaccessibleAddress=0000000100046E00
 [OUT] R0=0000000100046E00 R1=0000000000000200 R2=0000000100046E00 R3=0000000000000005
 [OUT] R4=000000000000000E R5=0000FFFF6DE4F8E0 R6=0000FFFEF40005B0 R7=0000000000000000
 [OUT] R8=0000000000000000 R9=0000000000000000 R10=0000000000000001 R11=0000000000000000
 [OUT] R12=0101010101010101 R13=000000000000270F R14=000000000000001F R15=00000000000020D0
 [OUT] R16=0000FFFF946200D8 R17=0000FFFF9535D358 R18=0000000000000001 R19=0000FFFEF4011B60
 [OUT] R20=0000000000000200 R21=0000000100046E00 R22=0000FFFEF4011C08 R23=0000FFFEF4011C08
 [OUT] R24=00000000001D1368 R25=0000FFFF95251F50 R26=0000000000000001 R27=0000FFFF94621B08
 [OUT] R28=0005FE14E522F540 R29=0000FFFF6DE4D850 R30=0000FFFF9535D380 R31=0000FFFF6DE4D850
 [OUT] PC=0000FFFF9535D040 SP=0000FFFF6DE4D850 PSTATE=0000000040001000
 [OUT] V0 952531f0b61f2fe1 (f: 3055497216.000000, d: -8.252167e-207)
 [OUT] V1 00000000b61f2fe1 (f: 3055497216.000000, d: 1.509616e-314)
 [OUT] V2 0000ffff233a1e11 (f: 591011328.000000, d: 1.390653e-309)
 [OUT] V3 bfd00ea348b88334 (f: 1220051712.000000, d: -2.508934e-01)
 [OUT] V4 bfdff8a542b00cd5 (f: 1118833920.000000, d: -4.995511e-01)
 [OUT] V5 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
 [OUT] V6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [OUT] V7 8020080280200802 (f: 2149582848.000000, d: -4.458850e-308)
 [OUT] V8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 4010040140100401 (f: 1074791424.000000, d: 4.003911e+00)
 [OUT] V17 0100000000000000 (f: 0.000000, d: 7.291122e-304)
 [OUT] V18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V19 3f9eb851eb851eb8 (f: 3951369984.000000, d: 3.000000e-02)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/lib64/libc.so.6
 [OUT] Module_base_address=0000FFFF952E0000 Symbol=strnlen
 [OUT] Symbol_address=0000FFFF9535D024
 [OUT] Target=2_90_20240123_607 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1a0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x847f84) [0xffff8fca7f84]
 [OUT] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x854ab0) [0xffff8fcb4ab0]
 [OUT] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x124904) [0xffff8f584904]
 [OUT] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x27cd4) [0xffff94c77cd4]
 [OUT] #4: function __kernel_rt_sigreturn+0 [0]
 [OUT] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e56fc) [0xffff8f6456fc]
 [OUT] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5da0) [0xffff8f645da0]
 [OUT] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5eb0) [0xffff8f645eb0]
 [OUT] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e6038) [0xffff8f646038]
 [OUT] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5918) [0xffff8f645918]
 [OUT] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1d1a8c) [0xffff8f631a8c]
 [OUT] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1d0a5c) [0xffff8f630a5c]
 [OUT] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1135b0) [0xffff8f5735b0]
 [OUT] #13: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4aa04c) [0xffff8f90a04c]
 [OUT] #14: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4ac2d4) [0xffff8f90c2d4]
 [OUT] #15: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4c9680) [0xffff8f929680]
 [OUT] #16: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8ae6c8) [0xffff8fd0e6c8]
 [OUT] #17: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4c9680) [0xffff8f929680]
 [OUT] #18: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1006ac) [0xffff8f5606ac]
 [OUT] #19: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d6670) [0xffff8f936670]
 [OUT] #20: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d72e4) [0xffff8f9372e4]
 [OUT] #21: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d2a94) [0xffff8f932a94]
 [OUT] #22: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4fb42c) [0xffff8f95b42c]
 [OUT] #23: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13bc00) [0xffff8f59bc00]
 [OUT] #24: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13ca28) [0xffff8f59ca28]
 [OUT] #25: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x287ec) [0xffff94c787ec]
 [OUT] #26: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13a738) [0xffff8f59a738]
 [OUT] #27: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13abe8) [0xffff8f59abe8]
 [OUT] #28: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x139970) [0xffff8f599970]
 [OUT] #29: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x139e2c) [0xffff8f599e2c]
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x0005ff04
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF94D11E44 Handler2=0000FFFF94C77B60 InaccessibleAddress=0000000000000008
 [OUT] R0=0000FFFF6E18DC40 R1=0000000000000000 R2=0000FFFF742C7AD8 R3=0000FFFF2E7F8DC0
 [OUT] R4=0000FFFF8F6451F4 R5=0000FFFF8FF95FD8 R6=0000FFFF941EAA59 R7=0000000072687339
 [OUT] R8=000000000A6CDCA8 R9=0000000000000018 R10=0022041FE89DA093 R11=0000265A2FEDC500
 [OUT] R12=000000007FFFFFFF R13=0000FFFF742C7B00 R14=0000000000000000 R15=0000000000000001
 [OUT] R16=0000FFFF94620010 R17=0000FFFF95364950 R18=0000000000000000 R19=0000FFFF6E18DC40
 [OUT] R20=0000000000000000 R21=0000FFFF742C7B88 R22=0000FFFF2E7F8DC0 R23=0000FFFF742C7B88
 [OUT] R24=0000FFFF742C7B88 R25=0000FFFF2E7F8DC0 R26=00000000000CC0A0 R27=0000000000000000
 [OUT] R28=0000000000000000 R29=0000FFFF742CCF10 R30=0000FFFF8F645DA0 R31=0000FFFF742C7AA0
 [OUT] PC=0000FFFF8F6456FC SP=0000FFFF742C7AA0 PSTATE=0000000000001000
 [OUT] V0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V1 0000013f000000b0 (f: 176.000000, d: 6.769167e-312)
 [OUT] V2 00000130000000a1 (f: 161.000000, d: 6.450867e-312)
 [OUT] V3 ffffffffffffffff (f: 4294967296.000000, d: -nan)
 [OUT] V4 0000000200000001 (f: 1.000000, d: 4.243992e-314)
 [OUT] V5 0000001000000010 (f: 16.000000, d: 3.395193e-313)
 [OUT] V6 0000001000000018 (f: 24.000000, d: 3.395193e-313)
 [OUT] V7 0000004000000040 (f: 64.000000, d: 1.358077e-312)
 [OUT] V8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 4010040140100401 (f: 1074791424.000000, d: 4.003911e+00)
 [OUT] V17 aa00000000108800 (f: 1083392.000000, d: -2.180075e-106)
 [OUT] V18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
 [OUT] Module_base_address=0000FFFF8F460000
 [OUT] 
 [OUT] Method_being_compiled=java/nio/Buffer.<init>(IIII)V
 [OUT] Target=2_90_20240123_607 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1a0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] strnlen+0x1c (0x0000FFFF9535D040 [libc.so.6+0x7d040])
 [OUT] strncpy+0x28 (0x0000FFFF9535D380 [libc.so.6+0x7d380])
 [OUT] getTrcBuf+0x2f4 (0x0000FFFF945DEC94 [libj9trc29.so+0xec94])
 [OUT] traceV+0x170 (0x0000FFFF945DF380 [libj9trc29.so+0xf380])
 [OUT] doTracePoint+0x768 (0x0000FFFF945E1258 [libj9trc29.so+0x11258])
 [OUT] omrTrace+0x6c (0x0000FFFF945E17BC [libj9trc29.so+0x117bc])
 [OUT] walkStackFrames+0x11c (0x0000FFFF94D4F9FC [libj9vm29.so+0x6f9fc])
 [OUT] JVM_GetCallerClass_Impl+0xbc (0x0000FFFF940AA0CC [libjclse29.so+0x5a0cc])
 [OUT]  (0x0000FFFF742D0758 [<unknown>+0x0])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/22 22:06:55 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220655.3605248.0001.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220655.3605248.0001.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220655.3605248.0002.txt' in response to an event
 [OUT] _ZN16TR_J9SharedCache28romclassMatchesCachedVersionEP10J9ROMClassRPmS2_+0x2c (0x0000FFFF8F6456FC [libj9jit29.so+0x1e56fc])
 [OUT] _ZN16TR_J9SharedCache30validateInterfacesInClassChainEP19TR_OpaqueClassBlockRPmS2_+0x70 (0x0000FFFF8F645DA0 [libj9jit29.so+0x1e5da0])
 [OUT] _ZN16TR_J9SharedCache18validateClassChainEP10J9ROMClassP19TR_OpaqueClassBlockRPmS4_+0xc0 (0x0000FFFF8F645EB0 [libj9jit29.so+0x1e5eb0])
 [OUT] _ZN16TR_J9SharedCache25classMatchesCachedVersionEP7J9ClassPm+0x138 (0x0000FFFF8F646038 [libj9jit29.so+0x1e6038])
 [OUT] _ZN16TR_J9SharedCache13rememberClassEP7J9ClassPPK24AOTCacheClassChainRecordb+0xf8 (0x0000FFFF8F645918 [libj9jit29.so+0x1e5918])
 [OUT] _ZN30TR_ResolvedRelocatableJ9Method32storeValidationRecordIfNecessaryEPN2TR11CompilationEP14J9ConstantPooli31TR_ExternalRelocationTargetKindP8J9MethodP7J9Class+0x178 (0x0000FFFF8F631A8C [libj9jit29.so+0x1d1a8c])
 [OUT] _ZThn88_N30TR_ResolvedRelocatableJ9Method24getClassFromConstantPoolEPN2TR11CompilationEjb+0x13c (0x0000FFFF8F630A5C [libj9jit29.so+0x1d0a5c])
 [OUT] _ZN2J911Compilation17canAllocateInlineEPN2TR4NodeERP19TR_OpaqueClassBlock+0xe0 (0x0000FFFF8F5735B0 [libj9jit29.so+0x1135b0])
 [OUT] _ZN2J95ARM6413TreeEvaluator14VMnewEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x9c (0x0000FFFF8F90A04C [libj9jit29.so+0x4aa04c])
 [OUT] _ZN2J95ARM6413TreeEvaluator18newObjectEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x14 (0x0000FFFF8F90C2D4 [libj9jit29.so+0x4ac2d4])
 [OUT] _ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x50 (0x0000FFFF8F929680 [libj9jit29.so+0x4c9680])
 [OUT] _ZN3OMR5ARM6413TreeEvaluator16treetopEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x24 (0x0000FFFF8FD0E6C8 [libj9jit29.so+0x8ae6c8])
 [OUT] _ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x50 (0x0000FFFF8F929680 [libj9jit29.so+0x4c9680])
 [OUT] _ZN2J913CodeGenerator22doInstructionSelectionEv+0x97c (0x0000FFFF8F5606AC [libj9jit29.so+0x1006ac])
 [OUT] _ZN3OMR12CodeGenPhase32performInstructionSelectionPhaseEPN2TR13CodeGeneratorEPNS1_12CodeGenPhaseE+0x70 (0x0000FFFF8F936670 [libj9jit29.so+0x4d6670])
 [OUT] _ZN3OMR12CodeGenPhase10performAllEv+0xf4 (0x0000FFFF8F9372E4 [libj9jit29.so+0x4d72e4])
 [OUT] _ZN3OMR13CodeGenerator12generateCodeEv+0x60 (0x0000FFFF8F932A94 [libj9jit29.so+0x4d2a94])
 [OUT] _ZN3OMR11Compilation7compileEv+0x65c (0x0000FFFF8F95B42C [libj9jit29.so+0x4fb42c])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x40c (0x0000FFFF8F59BC00 [libj9jit29.so+0x13bc00])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x2c8 (0x0000FFFF8F59CA28 [libj9jit29.so+0x13ca28])
 [OUT] omrsig_protect+0x21c (0x0000FFFF94C787EC [libj9prt29.so+0x287ec])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x2c4 (0x0000FFFF8F59A738 [libj9jit29.so+0x13a738])
 [OUT] _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x118 (0x0000FFFF8F59ABE8 [libj9jit29.so+0x13abe8])
 [OUT] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x2e0 (0x0000FFFF8F599970 [libj9jit29.so+0x139970])
 [OUT] _ZN2TR24CompilationInfoPerThread3runEv+0x4c (0x0000FFFF8F599E2C [libj9jit29.so+0x139e2c])
 [OUT] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x7c (0x0000FFFF8F599EDC [libj9jit29.so+0x139edc])
 [OUT] omrsig_protect+0x21c (0x0000FFFF94C787EC [libj9prt29.so+0x287ec])
 [OUT] _Z21compilationThreadProcPv+0x148 (0x0000FFFF8F59A2A8 [libj9jit29.so+0x13a2a8])
 [OUT] thread_wrapper+0xcc (0x0000FFFF94C173BC [libj9thr29.so+0x73bc])
 [OUT] start_thread+0x188 (0x0000FFFF954C7898 [libpthread.so.0+0x7898])
 [OUT] thread_start+0xc (0x0000FFFF953B1DDC [libc.so.6+0xd1ddc])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/22 22:07:05 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220705.3605248.0003.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220705.3605248.0003.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220705.3605248.0004.txt' in response to an event
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220705.3605248.0004.txt
 [OUT] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/Snap.20240122.220705.3605248.0005.trc' in response to an event
 [OUT] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/Snap.20240122.220705.3605248.0005.trc
 [OUT] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/jitdump.20240122.220705.3605248.0006.dmp' in response to an event
 [OUT] JVMDUMP051I JIT dump occurred in 'JIT Compilation Thread-000' thread 0x0000000000015100
 [OUT] JVMDUMP049I JIT dump notified all waiting threads of the current method to be compiled
 [OUT] JVMDUMP054I JIT dump is tracing the IL of the method on the crashed compilation thread
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220655.3605248.0002.txt
 [OUT] JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
 [OUT] JVMDUMP048I JIT dump method being compiled is an ordinary method
 [OUT] JVMDUMP053I JIT dump is recompiling java/nio/Buffer.<init>(IIII)V
 [OUT] JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
 [OUT] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/jitdump.20240122.220705.3605248.0006.dmp
 [OUT] JVMDUMP013I Processed dump event "gpf", detail "".
 [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
 [OUT] Removed test output files
 [OUT] finished script
>> Success condition was not found: [Output match: User requested Java dump using]
>> Success condition was not found: [Output match: TEST PASSED]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Checkpoint blocked because thread]
>> Required condition was not found: [Output match: Killed]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
pshipton commented 9 months ago

@TobiAjila pls help direct this.

pshipton commented 9 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/433 cmdLineTester_criu_nonPortableRestore_5 -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/433/functional_test_output.tar.gz

Testing: Create and Restore Criu Checkpoint Image once - MethodTypeDeadlockTest
Test start time: 2024/01/24 05:58:21 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/java " -Xgcpolicy:optthruput  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED" org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
Time spent starting: 4 milliseconds
Time spent executing: 3499 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xgcpolicy:optthruput  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
 [OUT] Pre-checkpoint
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x00000000
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF9D782F44 Handler2=0000FFFF9D6E7B60 InaccessibleAddress=0000412474736580
 [OUT] R0=0000000089383CC0 R1=0000000000000008 R2=0000000000000000 R3=0000000000000004
 [OUT] R4=0000FFFF4E87DAA0 R5=0000000000000000 R6=0000FFFF9D091B08 R7=FFFFFF80FFFFFFE0
 [OUT] R8=0000FFFF4E87DC80 R9=0000FFFF4E87DC80 R10=0000FFFF4E87DC60 R11=0000000000000011
 [OUT] R12=0000FFFF4E87DC80 R13=0000FFFF4E87DC80 R14=0000000000048108 R15=00000000890B06E8
 [OUT] R16=0000FFFF97F700B0 R17=0000FFFF9DDD5480 R18=0000FFFF4E87E300 R19=0000000000208468
 [OUT] R20=0000FFFF9D91FB40 R21=0000000000000000 R22=0000FFFF9D973810 R23=0000FFFF4E87DC80
 [OUT] R24=0000FFFF9D08F000 R25=00000000FFFFFFE0 R26=0000412474736554 R27=0000000000000004
 [OUT] R28=0000FFFF9DCC1F90 R29=0000FFFF4E87D870 R30=0000FFFF9D051258 R31=0000FFFF4E87D870
 [OUT] PC=0000FFFF9D04F2EC SP=0000FFFF4E87D870 PSTATE=0000000020001000
 [OUT] V0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V1 007200650073002e (f: 7536686.000000, d: 1.602190e-306)
 [OUT] V2 004100540045004d (f: 4522061.000000, d: 1.891455e-307)
 [OUT] V3 000000490000002d (f: 45.000000, d: 1.549057e-312)
 [OUT] V4 9728f76acff8b19e (f: 3489182208.000000, d: -4.174938e-197)
 [OUT] V5 00000000a64fe1c0 (f: 2790253056.000000, d: 1.378568e-314)
 [OUT] V6 94446f0194446f01 (f: 2487512832.000000, d: -4.855772e-211)
 [OUT] V7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
 [OUT] V8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 0000000044000000 (f: 1140850688.000000, d: 5.636551e-315)
 [OUT] V17 0000000400000000 (f: 0.000000, d: 8.487983e-314)
 [OUT] V18 0000000000000400 (f: 1024.000000, d: 5.059232e-321)
 [OUT] V19 3f9eb851eb851eb8 (f: 3951369984.000000, d: 3.000000e-02)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9trc29.so
 [OUT] Module_base_address=0000FFFF9D040000
 [OUT] Target=2_90_20240124_448 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1b0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] traceV+0xdc (0x0000FFFF9D04F2EC [libj9trc29.so+0xf2ec])
 [OUT] doTracePoint+0x768 (0x0000FFFF9D051258 [libj9trc29.so+0x11258])
 [OUT] omrTrace+0x6c (0x0000FFFF9D0517BC [libj9trc29.so+0x117bc])
 [OUT] walkStackFrames+0x94 (0x0000FFFF9D7C3014 [libj9vm29.so+0x73014])
 [OUT] Java_java_security_AccessController_getAccSnapshot+0x140 (0x0000FFFF97EE8DF0 [libjclse29.so+0x18df0])
 [OUT]  (0x0000FFFF7CFC34D0 [<unknown>+0x0])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/24 05:58:21 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/core.20240124.055821.2624260.0001.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/core.20240124.055821.2624260.0001.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/javacore.20240124.055821.2624260.0002.txt' in response to an event
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Jan 24 05:58:21 EST 2024, System.currentTimeMillis(): 1706093901922, System.nanoTime(): 1801422638095851
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/javacore.20240124.055821.2624260.0002.txt
 [OUT] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/Snap.20240124.055821.2624260.0003.trc' in response to an event
 [OUT] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/Snap.20240124.055821.2624260.0003.trc
 [OUT] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/jitdump.20240124.055821.2624260.0004.dmp' in response to an event
 [OUT] JVMDUMP051I JIT dump occurred in 'Thread-5' thread 0x0000000000208400
 [OUT] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/jitdump.20240124.055821.2624260.0004.dmp
 [OUT] JVMDUMP013I Processed dump event "gpf", detail "".
 [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
 [OUT] Removed test output files
 [OUT] finished script
>> Success condition was not found: [Output match: User requested Java dump using]
>> Success condition was not found: [Output match: TEST PASSED]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Checkpoint blocked because thread]
>> Required condition was not found: [Output match: Killed]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
tajila commented 9 months ago

This may be a regression.

@ThanHenderson can you please take a look at this

ThanHenderson commented 9 months ago

I reproduced a segmentation fault for cmdLineTester_criu_nonPortableRestore_8 on my personal x86-64_linux fyre dev machine. It is very intermittent with an observed 2/1100 failures. These failures had different stack traces than what is reported here.

For cmdLineTester_criu_nonPortableRestore_5, I tried running 1000 iterations twice, both times an iteration hung before completing all iterations.

I'll investigate to see if this is indeed a recent regression or just such an intermittent problem that it hadn't shown up for a while.

ThanHenderson commented 9 months ago

This is an issue with the tracing mechanism, and doesn't show up when using -Xtrace:none (passed >10,000 iterations successfully). I'm unsure exactly under which context it is happening, but I've narrowed it down to that. I'll have more insight later today.

ThanHenderson commented 8 months ago

This is what a normal UtThreadData looks like

UtThreadData at 0x7fffcc004eb0 {
  Fields for UtThreadData:
        0x0: struct UtDataHeader header = !utdataheader 0x00007FFFCC004EB0
        0x10: const void* id = !j9x 0x000000000001E600
        0x18: const void* synonym1 = !j9x 0x00007FFFF00D2E90
        0x20: const void* synonym2 = !j9x 0x000000000001F020
        0x28: const U8* name = !j9x 0x00007FFFCC004F50 // "(unnamed thread)"
        0x30: U8 currentOutputMask = 0x02 (2)
        0x38: struct UtTraceBuffer* trcBuf = !uttracebuffer 0x0000000000000000
        0x40: void* external = !j9x 0x0000000000000000
        0x48: I32 suspendResume = 0x00000000 (0)
        0x4c: I32 recursion = 0x00000001 (1)
        0x50: I32 indent = 0x00000000 (0)
}

But in the failing cases the UtThreadData is initialized as the first argument here https://github.com/eclipse-openj9/openj9/blob/5d03ee9d07d21881fae750fa5c69565325ffa364/runtime/rastrace/trclog.c#L2140 produces garbage data, similar to what is seen below, which causes problems at various points downstream in the tracing pipeline

UtThreadData at 0x7ffde81b0 {
  Fields for UtThreadData:
        0x0: struct UtDataHeader header = !utdataheader 0x00000007FFDE81B0
        0x10: const void* id = !j9x 0x07130003000A1700
        0x18: const void* synonym1 = !j9x 0x0100011500071400
        0x20: const void* synonym2 = !j9x 0x0600014901000178
        0x28: const U8* name = !j9x 0x00013E74696E693C
        0x30: U8 currentOutputMask = 0x03 (3)
        0x38: struct UtTraceBuffer* trcBuf = !uttracebuffer 0x694C0F000165646F
        0x40: void* external = !j9x 0x7265626D754E656E
        0x48: I32 suspendResume = 0x6C626154 (1818386772)
        0x4c: I32 recursion = 0x12000165 (301990245)
        0x50: I32 indent = 0x61636F4C (1633906508)
}
ThanHenderson commented 8 months ago

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified https://github.com/eclipse-openj9/openj9/pull/18745 as the commit that introduces this regression.

JasonFengJ9 commented 8 months ago

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified https://github.com/eclipse-openj9/openj9/pull/18745 as the commit that introduces this regression.

I will have a look it.

ThanHenderson commented 8 months ago

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified #18745 as the commit that introduces this regression.

This is actually not the case. After that commit, they just seem to happen more frequently. Running the commit before for ~50,000 iterations I observed the failure 4 times.

JasonFengJ9 commented 8 months ago

The MethodTypeDeadlockTest crash might be related to the two UT_MODULE_LOADED(J9_UTINTERFACE_FROM_VM(vm)) calls within j9vm module. One is https://github.com/eclipse-openj9/openj9/blob/43f37078c4fc4229e6a42e19893ffc080c61f90a/runtime/vm/jvminit.c#L2878 The other is https://github.com/eclipse-openj9/openj9/blob/43f37078c4fc4229e6a42e19893ffc080c61f90a/runtime/vm/CRIUHelpers.cpp#L903 which should be removed at

I was able to reproduce the crash in a fyre x64 Linux image ~10/1000 runs, removing the UT_MODULE_LOADED() call within CRIUHelpers.cpp hasn't crashed in 2000+ runs.

@ThanHenderson could you try a build in your test environment?