search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

DaaLoadTest_daa1 tests - **FAILED** Process DLT has hung #13477

Open JasonFengJ9 opened 3 years ago

JasonFengJ9 commented 3 years ago

Failure link

From an internal build Test_openjdk11_j9_special.system_ppc64le_linux_testList_2/111/ (ubu18le-rt1-4):

[2021-09-09T02:19:34.473Z] java version "11.0.13-beta" 2021-10-19
[2021-09-09T02:19:34.473Z] IBM Semeru Runtime Certified Edition 11.0.12.0 (build 11.0.13-beta+5-202109090107)
[2021-09-09T02:19:34.473Z] Eclipse OpenJ9 VM 11.0.12.0 (build master-92ccf02b2, JRE 11 Linux ppc64le-64-Bit Compressed References 20210908_164 (JIT enabled, AOT enabled)
[2021-09-09T02:19:34.473Z] OpenJ9   - 92ccf02b2
[2021-09-09T02:19:34.473Z] OMR      - 680b68120
[2021-09-09T02:19:34.473Z] JCL      - e851d14114 based on jdk-11.0.13+5)

Re-run the Grinder

Optional info

Failure output (captured from console output)

[2021-09-09T02:19:28.973Z] Running test DaaLoadTest_daa1_special_5m_4 ...
[2021-09-09T02:19:28.973Z] ===============================================
[2021-09-09T02:19:28.973Z] DaaLoadTest_daa1_special_5m_4 Start Time: Wed Sep  8 19:19:27 2021 Epoch Time (ms): 1631153967861
[2021-09-09T02:19:28.974Z] "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_testList_2/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_testList_2/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
[2021-09-09T02:19:28.974Z] JVMSHRC005I No shared class caches available
[2021-09-09T02:19:28.974Z] JVMSHRC005I No shared class caches available
[2021-09-09T02:19:28.974Z] cache cleanup done
[2021-09-09T02:19:28.974Z] variation: Mode112
[2021-09-09T02:19:28.974Z] JVM_OPTIONS:  -Xgcpolicy:gencon -Xjit:count=0 -Xnocompressedrefs 

[2021-09-09T02:20:04.486Z] DLT 19:20:03.029 - Final test list:
[2021-09-09T02:20:04.486Z] DLT 19:20:03.129 -   0 ArbitraryJava[net.openj9.test.simple.ConvertDecimal invokeTest]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.130 -   1 ArbitraryJava[net.openj9.test.simple.MarshalUnmarshalBinary invokeTest]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.132 -   2 JUnit[net.openj9.test.arithmetics.TestArithmeticOperations]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.132 -   3 JUnit[net.openj9.test.arithmetics.TestArithmeticInline]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.132 -   4 JUnit[net.openj9.test.arithmetics.TestArithmetics]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   5 JUnit[net.openj9.test.arithmetics.TestComparisonEquals]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   6 JUnit[net.openj9.test.arithmetics.TestPDComparisons]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   7 JUnit[net.openj9.test.arithmetics.TestPerformance]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   8 JUnit[net.openj9.test.arithmetics.TestSubExceptions]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   9 JUnit[net.openj9.test.arithmetics.TestValidityChecking]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.133 -   10 JUnit[net.openj9.test.binaryData.LongIntegerComparison]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   11 JUnit[net.openj9.test.binaryData.TestOptimizer]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   12 JUnit[net.openj9.test.decimals.TestBD2PD2BD]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   13 JUnit[net.openj9.test.decimals.TestDecimalData]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   14 JUnit[net.openj9.test.decimals.TestDecimalData2]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   15 JUnit[net.openj9.test.PD2Primitive.TestPD2Primitives2PD]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   16 JUnit[net.openj9.test.PDMoveShifts.TestShiftsAndConvert]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   17 JUnit[net.openj9.test.PDMoveShifts.TestPDMove]  Weighting=1 
[2021-09-09T02:20:04.486Z] DLT 19:20:03.134 -   18 ArbitraryJava[net.openj9.test.PDMoveShifts.ShiftTestRunner invokeTest]  Weighting=1 
[2021-09-09T02:20:06.734Z] DLT 19:20:05.197 - Starting thread. Suite=0 thread=0
[2021-09-09T02:20:06.734Z] DLT 19:20:05.248 - Starting thread. Suite=0 thread=1
[2021-09-09T02:24:45.299Z] STF 19:24:33.408 - Heartbeat: Process DLT is still running
[2021-09-09T02:29:45.506Z] STF 19:29:33.130 - Heartbeat: Process DLT is still running
[2021-09-09T02:34:46.648Z] STF 19:34:33.380 - Heartbeat: Process DLT is still running
[2021-09-09T02:39:33.486Z] DLT 19:39:22.522 - Completed 100.0%. Number of tests started=2
[2021-09-09T02:39:33.486Z] DLT 19:39:22.523 - **POSSIBLE HANG DETECTED**
[2021-09-09T02:39:33.487Z] STF 19:39:22.601 - **FAILED** Process DLT has hung
[2021-09-09T02:39:33.487Z] STF 19:39:22.601 - Collecting dumps for: DLT
[2021-09-09T02:39:33.487Z] STF 19:39:22.601 - Sending SIG 3 to the java process to generate a javacore

[2021-09-09T02:40:55.178Z] STF 19:40:52.949 - =====================   R E S U L T S   =====================
[2021-09-09T02:40:55.178Z] STF 19:40:52.949 - Stage results:
[2021-09-09T02:40:55.178Z] STF 19:40:52.949 -   setUp:     pass
[2021-09-09T02:40:55.178Z] STF 19:40:52.949 -   execute:  *fail*
[2021-09-09T02:40:55.178Z] STF 19:40:52.949 -   teardown:  pass
[2021-09-09T02:40:55.178Z] STF 19:40:52.949 - 
[2021-09-09T02:40:55.178Z] STF 19:40:52.950 - Overall result: **FAILED**
[2021-09-09T02:40:55.178Z] 
[2021-09-09T02:40:55.178Z] DaaLoadTest_daa1_special_5m_4_FAILED
pshipton commented 1 year ago

@JasonFengJ9 pls ignore DaaLoadTest mode 24 hangs on Windows. https://github.com/eclipse-openj9/openj9/issues/13477#issuecomment-1485424482

JasonFengJ9 commented 1 year ago

JDK21 aarch64_mac milestone 0(macaarch64rt2)

[2023-09-22T19:43:32.558Z] variation: -Xgcpolicy:gencon -Xgc:concurrentScavenge
[2023-09-22T19:43:32.558Z] JVM_OPTIONS:  -Xgcpolicy:gencon -Xgc:concurrentScavenge 

[2023-09-22T19:59:38.816Z] STF 15:59:13.231 - Heartbeat: Process DLT is still running
[2023-09-22T19:59:38.816Z] DLT 15:59:13.244 - Completed 100.0%. Number of tests started=924
[2023-09-22T19:59:38.816Z] DLT 15:59:13.244 - **POSSIBLE HANG DETECTED**

[2023-09-22T19:59:44.913Z] STF 15:59:44.326 - Overall result: **FAILED**
[2023-09-22T19:59:44.913Z] -----------------------------------
[2023-09-22T19:59:44.913Z] DaaLoadTest_daa1_CS_5m_0_FAILED
hzongaro commented 1 year ago

Moving this to the backlog for now, but I will try to spend some time sorting out whether there are distinct JIT problems here that can be tracked and prioritized as separate issues.

JasonFengJ9 commented 7 months ago

JDK11 aarch64_mac(macaarch64rt4)

[2024-03-26T19:20:56.352Z] variation: Mode104
[2024-03-26T19:20:56.352Z] JVM_OPTIONS:  -Xgcpolicy:optthruput -Xjit:count=0,optlevel=hot,gcOnResolve,rtResolve -Xnocompressedrefs 

[2024-03-26T19:50:31.743Z] STF 15:50:01.429 - Heartbeat: Process DLT is still running
[2024-03-26T19:50:31.743Z] DLT 15:50:01.841 - Completed 6.7%. Number of tests started=140
[2024-03-26T19:50:31.743Z] DLT 15:50:01.842 - **POSSIBLE HANG DETECTED**
[2024-03-26T19:50:31.743Z] STF 15:50:01.940 - **FAILED** Process DLT has hung

[2024-03-26T19:50:32.199Z] DLT 15:50:04.677 - Load test completed
[2024-03-26T19:50:32.199Z] DLT 15:50:04.677 -   Ran     : 140
[2024-03-26T19:50:32.199Z] DLT 15:50:04.677 -   Passed  : 140
[2024-03-26T19:50:32.199Z] DLT 15:50:04.677 -   Failed  : 0
[2024-03-26T19:50:32.199Z] DLT 15:50:04.677 -   Result  : PASSED
[2024-03-26T19:50:32.199Z] DLT stderr JVMDUMP039I Processing dump event "user", detail "" at 2024/03/26 15:50:01 - please wait.
[2024-03-26T19:50:32.199Z] DLT stderr JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk11_j9_special.system_aarch64_mac_testList_3/aqa-tests/TKG/output_17114786051176/DaaLoadTest_daa1_special_5m_2/20240326-152056-DaaLoadTest/results/core.20240326.155001.32750.0001.dmp' in response to an event
[2024-03-26T19:50:32.199Z] STF 15:50:31.937 - Monitoring Report Summary:
[2024-03-26T19:50:32.199Z] STF 15:50:31.937 -   o Process DLT has hung
[2024-03-26T19:50:32.199Z] STF 15:50:31.938 - Killing processes: DLT
[2024-03-26T19:50:32.199Z] STF 15:50:31.938 -   o Process DLT pid 32750 is not running
[2024-03-26T19:50:32.199Z] **FAILED** at step 1 (Run daa load test). Expected return value=0 Actual=1 at /Users/jenkins/workspace/Test_openjdk11_j9_special.system_aarch64_mac_testList_3/aqa-tests/TKG/../TKG/output_17114786051176/DaaLoadTest_daa1_special_5m_2/20240326-152056-DaaLoadTest/execute.pl line 96.
[2024-03-26T19:50:32.199Z] STF 15:50:31.968 - **FAILED** execute script failed. Expected return value=0 Actual=1
[2024-03-26T19:50:32.199Z] STF 15:50:31.968 - 
[2024-03-26T19:50:32.199Z] STF 15:50:31.968 - ====================   T E A R D O W N   ====================
[2024-03-26T19:50:32.199Z] STF 15:50:31.968 - Running teardown: perl /Users/jenkins/workspace/Test_openjdk11_j9_special.system_aarch64_mac_testList_3/aqa-tests/TKG/../TKG/output_17114786051176/DaaLoadTest_daa1_special_5m_2/20240326-152056-DaaLoadTest/tearDown.pl
[2024-03-26T19:50:32.199Z] STF 15:50:32.035 - TEARDOWN stage completed
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 - 
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 - =====================   R E S U L T S   =====================
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 - Stage results:
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 -   setUp:     pass
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 -   execute:  *fail*
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 -   teardown:  pass
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 - 
[2024-03-26T19:50:32.199Z] STF 15:50:32.038 - Overall result: **FAILED**
[2024-03-26T19:50:32.199Z] -----------------------------------
[2024-03-26T19:50:32.199Z] DaaLoadTest_daa1_special_5m_2_FAILED

Another similar one DaaLoadTest_daa1_special_5m_24_FAILED at same machine.

JasonFengJ9 commented 6 months ago

JDK22 x86-64_windows(win16x64rt-1-8)

[2024-05-03T19:01:23.620Z] variation: Mode107-OSRG
[2024-05-03T19:01:24.002Z] JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation 

[2024-05-03T19:19:26.739Z] DLT 15:19:24.271 - Completed 100.0%. Number of tests started=108 (+0)
[2024-05-03T19:19:36.312Z] DLT 15:19:35.443 - **POSSIBLE HANG DETECTED**
[2024-05-03T19:19:36.312Z] STF 15:19:35.676 - **FAILED** Process DLT has hung
[2024-05-03T19:19:36.312Z] STF 15:19:35.677 - Collecting dumps for: DLT

[2024-05-03T19:21:18.058Z] DaaLoadTest_daa1_special_5m_24_FAILED
JasonFengJ9 commented 1 month ago

openjdk21_j9_special.system_x86-64_windows(win19x86-rtp-rt5-1)

[2024-10-08T17:07:40.206Z] variation: Mode107-OSRG
[2024-10-08T17:07:40.206Z] JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation 

[2024-10-08T17:26:55.266Z] DLT 10:26:51.806 - Completed 100.0%. Number of tests started=47 (+0)
[2024-10-08T17:27:11.998Z] DLT 10:27:04.587 - **POSSIBLE HANG DETECTED**
[2024-10-08T17:27:11.998Z] STF 10:27:04.853 - **FAILED** Process DLT has hung
[2024-10-08T17:27:11.998Z] STF 10:27:04.853 - Collecting dumps for: DLT

[2024-10-08T17:28:42.649Z] DaaLoadTest_daa1_special_5m_24_FAILED