search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

Internal JITServer test failures #14538

Open cjjdespres opened 2 years ago

cjjdespres commented 2 years ago

This thread collects various failures in internal JITServer tests that have been observed recently, for platforms other than Z. I'll be going through them to see if they can be reproduced without JITServer enabled.

Unable to reproduce

Resolved

cjjdespres commented 2 years ago

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/185/, failure in testDefaultDisclaimMemory_3, with console output

testDefaultDisclaimMemory_3 Start Time: Sat Feb 12 02:28:27 2022 Epoch Time (ms): 1644661707905
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
[IncludeExcludeTestAnnotationTransformer] [INFO] EXCLUDE_FILE environment variable: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt,/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/latest_exclude_17.txt
[IncludeExcludeTestAnnotationTransformer] [INFO] Processing exclude file: /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TestConfig/resources/excludes/feature_ojdkmh_exclude.txt
...
... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
...

Invalid class in objectPtr=00007F82A63115A0
10:28:29.463 0x7f823c003a00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK17_x86-64_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:1258: ((false))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/12 02:28:29 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/core.20220212.022829.32526.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/javacore.20220212.022829.32526.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16446556174205/testDefaultDisclaimMemory_3/Snap.20220212.022829.32526.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".

testDefaultDisclaimMemory_3_FAILED

Seems to be related to https://github.com/eclipse-openj9/openj9/issues/14396, and shows up without JITServer enabled.

cjjdespres commented 2 years ago

From build job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/, failure in cmdLineTester_classesdbgddrext_1, with excerpted console output

 ===============================================
 Running test cmdLineTester_classesdbgddrext_1 ...
 ===============================================
 cmdLineTester_classesdbgddrext_1 Start Time: Sun Feb 13 01:17:51 2022 Epoch Time (ms): 1644743871414
 variation: Mode610
 JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon
 Start SysV clean up for user:all
 Clean up is finished:
    Shared Memory Segments Removed:0
    Shared Semaphores Removed:1
 Parsing exclude list...
   add excludes for [all]
   add excludes for [linux_ppc-64]
 DONE in 0 ms.

 No tests are excluded.

 *** Starting test suite: J9 shared cache dbgext Tests ***
 Testing: Create new shared cache with partition
 Test start time: 2022/02/13 01:17:52 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xaot:forceaot,count=0,disableAsyncCompilation -Xshareclasses:name=shareddbgext,reset,modified=mod1
 Time spent starting: 90 milliseconds
 Time spent executing: 10087 milliseconds
 Test result: PASSED

 Executing command: rm -f j9core.dmp

 Testing: Create core file
 Test start time: 2022/02/13 01:18:02 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xmx4m -Xaot:forceaot,count=0 -Xshareclasses:name=shareddbgext -cp /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/utils/utils.jar -Xdump:system:file=j9core.dmp,events=vmstop org.openj9.test.ivj.Hanoi 2
 Time spent starting: 8 milliseconds
 Time spent executing: 13650 milliseconds
 Test result: PASSED

 Testing: Remove shared cache
 Test start time: 2022/02/13 01:18:16 Pacific Standard Time
 Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=shareddbgext,destroy
 Time spent starting: 19 milliseconds
 Time spent executing: 106 milliseconds
 Test result: FAILED
  [ERR] JVMSHRC023E Cache does not exist
  [ERR] JVMSHRC023E Cache does not exist
 >> Success condition was not found: [Output match: destroyed]

I haven't been able to reproduce this with the grinders that I've run.

cjjdespres commented 2 years ago

From build job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/254/, failure in ParallelStreamsLoadTest_special_J9_11 with console output

===============================================
Running test ParallelStreamsLoadTest_special_J9_11 ...
===============================================
ParallelStreamsLoadTest_special_J9_11 Start Time: Sun Feb 13 03:52:10 2022 Epoch Time (ms): 1644753130302
variation: Mode501
JVM_OPTIONS: -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
STF 03:52:10.589 - =========================   S T F   =========================
systemtest-prereqs has been processed, and set to: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqsRetrieving amount of free space on drive containing /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11
There is 204283 Mb free
STF 03:52:10.607 - ==================   G E N E R A T I O N   ==================
STF 03:52:10.609 - Checking JVM: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image
STF 03:52:10.610 - Starting process to generate scripts: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java  -Dlog4j.skipJansi=true -Djava.system.class.loader=net.adoptopenjdk.stf.runner.StfClassLoader -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../../jvmtest/system/lambdaLoadTest/..//systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/scripts/../bin net.adoptopenjdk.stf.runner.StfRunner -properties "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/stf_parameters.properties, , /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/config/stf.properties" -testDir "/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest"
GEN 03:52:13.980 - Found test. Project: 'openjdk.test.load' class: 'ParallelStreamsLoadTest.class' Dir: '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin'
GEN 03:52:14.053 - Found test. Project: 'openjdk.test.load' class: 'net.adoptopenjdk.stf.ParallelStreamsLoadTest'
GEN Classpath directories used by project 'openjdk.test.load':
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar
GEN   /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar
GEN 03:52:14.606 - Using Mode NoOptions. Values = ''
GEN 03:52:14.750 -
GEN 03:52:14.750 - Test command summary:
GEN 03:52:14.751 -   Step  Stage   Command           Description
GEN 03:52:14.752 -  -----+--------+-----------------+------------
GEN 03:52:14.752 -     1  execute  Run java          Run parallel streams load test
STF 03:52:14.803 -
STF 03:52:14.803 - Script generation completed
STF 03:52:14.803 -
STF 03:52:14.805 -
STF 03:52:14.805 - =======================   S E T U P   =======================
STF 03:52:14.805 - Running setup: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/setUp.pl
STF 03:52:14.932 - SETUP stage completed
STF 03:52:14.937 -
STF 03:52:14.937 - ====================   E X E C U T E -   ====================
STF 03:52:14.937 - Running execute: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl
STF 03:52:15.055 -
STF 03:52:15.055 - Java version
STF 03:52:15.055 - Running: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -version
openjdk version "11.0.15-internal" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15-internal+0-adhoc.jenkins.BuildJDK11ppc64lelinuxjitPersonal)
Eclipse OpenJ9 VM (build master-c462b8b9d0c, JRE 11 Linux ppc64le-64-Bit Compressed References 20220213_936 (JIT enabled, AOT enabled)
OpenJ9   - c462b8b9d0c
OMR      - ae234b55240
JCL      - 24e7209ed42 based on jdk-11.0.15+1)
STF 03:52:15.311 -
STF 03:52:15.312 - +------ Step 1 - Run parallel streams load test
STF 03:52:15.312 - | Run foreground process
STF 03:52:15.312 - |   Program:     /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
STF 03:52:15.312 - |   Mnemonic:    LT
STF 03:52:15.312 - |   Echo:        ECHO_ON
STF 03:52:15.312 - |   Expectation: CLEAN_RUN within 1h
STF 03:52:15.312 - |
STF 03:52:15.315 - Running command: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java -Xmx1024M -XX:+UseJITServer -Xjit -Xgcpolicy:balanced -Xnocompressedrefs -classpath /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.load/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/STF/stf.core/bin:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-api.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/log4j/log4j-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/junit.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/systemtest_prereqs/junit/hamcrest-core.jar:/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/jvmtest/system/aqa-systemtest/openjdk.test.lambdasAndStreams/bin net.adoptopenjdk.loadTest.LoadTest -resultsDir /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results -resultsPrefix 1.LT. -inactivityLimit 60m -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.parallelStreams.threadCount 2 -suite.parallelStreams.inventoryFile /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml -suite.parallelStreams.inventoryExcludeFile none -suite.parallelStreams.totalNumberTests 2 -suite.parallelStreams.selection sequential -suite.parallelStreams.seed -1 -suite.parallelStreams.repeatCount 1 -suite.parallelStreams.thinkingTime 0ms..0ms
STF 03:52:15.315 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stderr
STF 03:52:15.315 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.stdout
STF 03:52:15.337 - Monitoring processes: LT
LT  03:52:18.683 - Load test parameters
LT  03:52:18.692 -   Time limited         = false
LT  03:52:18.693 -   abortIfOutOfMemory   = true
LT  03:52:18.694 -   reportFailureLimit   = 1
LT  03:52:18.695 -   abortAtFailureLimit  = 10
LT  03:52:18.698 -   maxTotalLogFileSpace = 209715200
LT  03:52:18.699 -   maxSingleLogSize     = 8388608
LT  03:52:18.700 - Parameters for suite 0
LT  03:52:18.702 -   Suite name     = parallelStreams
LT  03:52:18.703 -   Number threads = 2
LT  03:52:18.704 -   Supplied seed  = -1
LT  03:52:18.705 -   Inventory file = /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory/openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.706 -   Exclude file   = none
LT  03:52:18.707 -   Number tests   = 2
LT  03:52:18.708 -   Repeat count   = 1
LT  03:52:18.709 -   Thinking time  = 0ms..0ms
LT  03:52:18.711 -   Selection mode = sequential
LT  03:52:18.726 - Parsing inventory file. Root=/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/1.LT.inventory File=openjdk.test.load/config/inventories/lambdasAndStreams/parallelStreams.xml
LT  03:52:18.740 - Final test list:
LT  03:52:18.745 -   0 JUnit[net.adoptopenjdk.test.streams.TestParallelStreamOperations]  Weighting=1
LT  03:52:18.899 - Starting thread. Suite=0 thread=0
LT  03:52:18.902 - Starting thread. Suite=0 thread=1
LT  03:52:38.936 - Completed 100.0%. Number of tests started=2
LT  03:52:58.971 - Completed 100.0%. Number of tests started=2 (+0)
LT  stderr 11:53:18.527 0x7ff830002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr 11:53:18.527 0x7ff83c002400    j9mm.107    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64le_linux_jit_Personal/openj9/runtime/gc_vlhgc/CopyForwardScheme.cpp:3577: ((false && (isObjectInEvacuateMemory(object))))
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2022/02/13 03:53:18 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp' in response to an event
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133622.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt' in response to an event
STF 03:53:22.925 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
STF 03:53:22.926 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0001.dmp
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp' in response to an event
LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0003.txt
LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.3133632.
LT  stderr
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt' in response to an event
LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc' in response to an event
STF 03:53:31.952 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
STF 03:53:31.953 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/javacore.20220213.035318.3133542.0005.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/core.20220213.035318.3133542.0002.dmp
LT  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/results/Snap.20220213.035318.3133542.0004.trc
LT  stderr JVMDUMP013I Processed dump event "traceassert", detail "".
STF 03:53:38.476 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)
STF 03:53:38.476 - Monitoring Report Summary:
STF 03:53:38.476 -   o Process LT  has crashed unexpectedly
STF 03:53:38.477 - Killing processes: LT
STF 03:53:38.477 -   o Process LT  pid 3133542 is not running
**FAILED** at step 1 (Run parallel streams load test). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/execute.pl line 97.
STF 03:53:38.904 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 03:53:38.904 -
STF 03:53:38.904 - ====================   T E A R D O W N   ====================
STF 03:53:38.904 - Running teardown: perl /home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16447445428559/ParallelStreamsLoadTest_special_J9_11/20220213-035210-ParallelStreamsLoadTest/tearDown.pl
STF 03:53:39.027 - TEARDOWN stage completed
STF 03:53:39.030 -
STF 03:53:39.030 - =====================   R E S U L T S   =====================
STF 03:53:39.030 - Stage results:
STF 03:53:39.030 -   setUp:     pass
STF 03:53:39.030 -   execute:  *fail*
STF 03:53:39.030 -   teardown:  pass
STF 03:53:39.031 -
STF 03:53:39.031 - Overall result: **FAILED**

ParallelStreamsLoadTest_special_J9_11_FAILED

I haven't been able to reproduce this with the grinders that I've run.

cjjdespres commented 2 years ago

From build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/188/, failures in testSoftMxLocal_LP4k_3,testSoftMxDisclaimMemory_LP4k_3,testDefaultDisclaimMemory_3. I think this one is a duplicate of https://github.com/eclipse-openj9/openj9/issues/14396 as well, based on that thread.

cjjdespres commented 2 years ago

From build job/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/186/, failures in jdk_lang_0 and jdk_lang_1. I believe the relevant console output is

03:51:13  java.lang.Throwable
03:51:13    at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13    at Basic.fail(Basic.java:2817)
03:51:13    at Basic.fail(Basic.java:2818)
03:51:13    at Basic.equal(Basic.java:2824)
03:51:13    at Basic.realMain(Basic.java:1385)
03:51:13    at Basic.main(Basic.java:2827)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13    at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  >'=C:=\,OPENJ9_JAVA_COMMAND_LINE=/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/openjdkbinary/j2sdk-image/bin/java -XX:+DisplayVMOutputToStderr -classpath /home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/java/lang/ProcessBuilder/Basic_id1.d:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/ProcessBuilder:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/TKG/output_16450868001065/jdk_lang_1/work/classes/1/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_x86-64_linux_jit_Personal/jvmtest/openjdk/jtreg/lib/jtreg.jar Basic$JavaChild System.getenv(),'< not equal to '=C:=\,'
03:51:13  java.lang.Throwable
03:51:13    at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13    at Basic.fail(Basic.java:2817)
03:51:13    at Basic.fail(Basic.java:2818)
03:51:13    at Basic.equal(Basic.java:2824)
03:51:13    at Basic.realMain(Basic.java:1902)
03:51:13    at Basic.main(Basic.java:2827)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13    at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  Incorrect handling of envstrings containing NULs
03:51:13  java.lang.Throwable
03:51:13    at java.base/java.lang.Thread.dumpStack(Thread.java:560)
03:51:13    at Basic.fail(Basic.java:2817)
03:51:13    at Basic.fail(Basic.java:2818)
03:51:13    at Basic.check(Basic.java:2821)
03:51:13    at Basic.realMain(Basic.java:1956)
03:51:13    at Basic.main(Basic.java:2827)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13    at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  java.lang.AssertionError: Some tests failed
03:51:13    at Basic.main(Basic.java:2829)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:51:13    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
03:51:13    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:51:13    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
03:51:13    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
03:51:13    at java.base/java.lang.Thread.run(Thread.java:889)
03:51:13  
03:51:13  JavaTest Message: Test threw exception: java.lang.AssertionError: Some tests failed
03:51:13  JavaTest Message: shutting down test
03:51:13  
03:51:13  STATUS:Failed.`main' threw exception: java.lang.AssertionError: Some tests failed
cjjdespres commented 2 years ago

Builds job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_jit_Personal/265/ and job/Test_openjdk17_j9_sanity.openjdk_ppc64le_linux_jit_Personal/180/ fail in exactly the same way that https://github.com/eclipse-openj9/openj9/issues/14538#issuecomment-1043050057 does.

mpirvu commented 2 years ago

For easier tracking I suggest we have a list in the description of this issue with some brief comments about what we've learned. E.g. fails without JITServer too, already tracked by issue N, crash or not, etc. I am suggesting this because this issue is going to become very large, very soon :-)

cjjdespres commented 2 years ago

Definitely a good idea.

pshipton commented 2 years ago

The recent java/lang/ProcessBuilder/Basic.java failures are caused by changes merged yesterday and I expect they will be fixed by changes to the OpenJDK repos within the next hour. Fixes are under review now. https://github.com/eclipse-openj9/openj9/pull/14520 https://github.com/eclipse-openj9/openj9/pull/14528

pshipton commented 2 years ago

The fixes are merged now.

cjjdespres commented 2 years ago

The grinder https://hyc-runtimes-jenkins.swg-devops.com/view/Test_grinder/job/Grinder/21194/ reproduces the ParallelStreamsLoadTest_special_J9_11 failure without JITAAS.

cjjdespres commented 2 years ago

The build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/268/ has failures in:

I believe that testSCCMLTests2_0, testSCCMLTests4_0, testSCCMLTests4_1 are all being tracked by #5310. They fail because certain core files aren't created that should be. The rest all seem to fail in ways vaguely similar to those three tests. An excerpt from the (very large) output of cmdLineTester_pltest_j9sig_ext_0:

 [ERR] ----------------------------------------
 [ERR] dump tests
 [ERR] ----------------------------------------
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_verify_functiontable_slots
 [ERR] Ending test j9dump_verify_functiontable_slots
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_name
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_with_name
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  213: j9dump_test_create_dump_with_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252389 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR] 
 [ERR]      LastErrorNumber: -108
 [ERR]      LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_name
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_from_signal_handler
 [ERR]   calling j9dump_create with filename: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/j9dump_test_create_dump_from_signal_handler
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  311: j9dump_test_create_dump_from_signal_handler j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252395 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/cmdLineTester_pltest_j9sig_ext_0/core"
 [ERR]      LastErrorNumber: -108
 [ERR]      LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_from_signal_handler
 [ERR] 
 [ERR] 
 [ERR] Starting test j9dump_test_create_dump_with_NO_name
 [ERR]   calling j9dump_create with empty filename
 [ERR] JVMPORT030W 
 [ERR] /home/jenkins/workspace/Build_JDK8_x86-64_linux_jit_Personal/openj9/runtime/tests/port/j9dumpTest.c line  153: j9dump_test_create_dump_with_NO_name j9dump_create returned: 1, with filename: The core file created by child process with pid = 1252401 was not found. Expected to find core file with name "core"
 [ERR]      LastErrorNumber: -108
 [ERR]      LastErrorMessage: No such file or directory
 [ERR] 
 [ERR] Ending test j9dump_test_create_dump_with_NO_name
 [ERR] 
 [ERR] Dump test done, failures detected.

And an excerpt from testDDRExtJunit_MonitorsAndDeadlock5_1:

     [exec] JVMDUMP039I Processing dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration" at 2022/02/18 02:07:24 - please wait.
     [exec] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP' in response to an event
     [exec] JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.1275170.
     [exec] 
     [exec] HelperExceptionForCoreGeneration is thrown and caught successfuly.
     [exec] Forcefully terminating process.
     [exec] addTestJitHint adding hint to openj9/internal/tools/attach/target/AttachHandler$1.<clinit>
     [exec] addTestJitHint adding hint to java/util/Vector$Itr.<init>
     [exec] addTestJitHint adding hint to java/lang/Thread$State.values
     [exec] JVMDUMP012E Error in System dump: The core file created by child process with pid = 1275170 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/core"
     [exec] JVMDUMP013I Processed dump event "throw", detail "j9vm/test/corehelper/HelperExceptionForCoreGeneration".

TCK.destroy.cache:
     [echo] Destroying cache
     [echo] Running j9vm.test.corehelper.StackMapCoreGenerator to destroy the cache
     [echo] Using JVM : /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
     [echo] classname = "j9vm.test.corehelper.StackMapCoreGenerator"
     [echo] Java VM Args:
     [echo]   jvmarg = -Xshareclasses:name=ddrextjunitSCC,destroy
     [java] JVMSHRC806I Compressed references persistent shared cache "ddrextjunitSCC" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
     [java] Java Result: 1

TCK.run.tests.ddrext:
     [echo] Running the DDR Extension Test
     [java] openjdk version "1.8.0_332-internal"
     [java] OpenJDK Runtime Environment (build 1.8.0_332-internal-_2022_02_18_08_00-b00)
     [java] Eclipse OpenJ9 VM (build master-c0f5771, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20220218_965 (JIT enabled, AOT enabled)
     [java] OpenJ9   - c0f5771
     [java] OMR      - 64690f6
     [java] JCL      - bd2e6cf based on jdk8u332-b02)
     [java] 
     [java] [AutoRun] [INFO] Warning: /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP is not directory or file!
     [java] [AutoRun] [ERROR] No dump file under /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/../TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/DDREXT.J9CORE.DMP

BUILD FAILED
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:152: The following error occurred while executing this line:
/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16451745074894/testDDRExtJunit_MonitorsAndDeadlock5_1/tck_ddrext.xml:124: Java returned: 255

The only one that's a little different is cmdLineTester_shrcdbgddrext_1. That one starts with the failure:

Testing: Check whether dump file exist
Test start time: 2022/02/18 01:56:32 Pacific Standard Time
Running command: ls
Time spent starting: 3 milliseconds
Time spent executing: 18 milliseconds
Test result: FAILED
Output from test:
 [OUT] Calculation.class
 [OUT] Calculation.java
 [OUT] cmdlinetestresources.jar
 [OUT] com
 [OUT] Dump.class
 [OUT] Dump.java
 [OUT] j9vm
 [OUT] JavaRuntimeVersion.class
 [OUT] JavaRuntimeVersion.java
 [OUT] META-INF
 [OUT] SystemProperties.class
 [OUT] SystemProperties.java
 [OUT] VMBench
 [OUT] WithDebugInfo.class
 [OUT] WithDebugInfo.java
 [OUT] WithoutDebugInfo.class
 [OUT] WithoutDebugInfo.java
>> Success condition was not found: [Output match: j9core1.dmp]

and the subsequent tests fail because j9core1.dmp doesn't exist. But even here it's a problem with the creation of a core dump that causes the tests to fail.

In #5310 it is suspected that the failures are caused by machine issues. Perhaps that is the case here as well. The child build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal_testList_1/259/ had all the failures, running on computer ub20x64rt1-1.

cjjdespres commented 2 years ago

Related to these failures, I ran a subset of the failing tests a few times on the same machine in this run and got all failures, and ran the same tests on a few different automatically-allocated machines in this run and got all successes. So it does seem like a problem with ub20x64rt1-1.

cjjdespres commented 2 years ago

I tried to reproduce the cmdLineTester_classesdbgddrext_1 failure from job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/261/ again, including running on the same machine, and I still can't reproduce it. I think it may just be highly intermittent.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal/191/, failure in cmdLineTester_getPid_0, with output

cmdLineTester_getPid_0 Start Time: Sat Feb 19 02:32:37 2022 Epoch Time (ms): 1645266757691
variation: NoOptions
JVM_OPTIONS: -XX:+UseJITServer 
*** Starting test suite: getPid test ***
Testing: Compare RuntimeMXBean.getPid() with getting PID from system
Test start time: 2022/02/19 02:32:48 Pacific Standard Time
Running command: perl /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/getPidTest.pl "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/runtimemxbeanTests/runtimemxbeanTests.jar GetPid
Time spent starting: 88 milliseconds
Time spent executing: 5059 milliseconds
Test result: FAILED
Output from test:
 [OUT] FAIL: RuntimeMXBean.getPID() returned 0 instead of 31948
>> Success condition was not found: [Output match: PASS]
>> Failure condition was found: [Output match: FAIL]

Appears to be a duplicate of #13085.

Build job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_jit_Personal/185/ experienced exactly the same failure.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/269/, failure in testSoftMxDisclaimMemory_3. In addition, the build job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/193/ has a failure in testSoftMxDisclaimMemory_LP4k_3. Both have been experienced before and are being tracked elsewhere. I suppose I'll not report failures like these in future.

cjjdespres commented 2 years ago

From job/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal/265/, failure in SharedClassesAPI_1, with output

JVMT Sun Feb 20 02:26:54 2022
JVMT Incorrect number of caches found. Expected at least 4, found 2
JVMT Sun Feb 20 02:26:54 2022
JVMT Iterating over caches: ERROR
STF 02:26:55.620 - Monitoring Report Summary:
STF 02:26:55.620 -   o Process JVMT ended with the expected exit code (0)
**FAILED** at step 71 (Verify Classes Caches using JVMTI). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_extended.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_16453495033600/SharedClassesAPI_1/20220220-022207-SharedClassesAPI/execute.pl line 2534.

Version information:

04:19:38  openjdk version "1.8.0_332-internal"
04:19:38  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_19_23_57-b00)
04:19:38  Eclipse OpenJ9 VM (build master-ce1c984f065, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220220_921 (JIT enabled, AOT enabled)
04:19:38  OpenJ9   - ce1c984f065
04:19:38  OMR      - 64690f622a7
04:19:38  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Appears to be tracked by #13239.

cjjdespres commented 2 years ago

From build job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/274/, failures in

These have a moderate overlap with the failures in this build. It looks like they all fail because a core dump file is not created, so they all fall under #5310 if I'm right about the connection between that issue and the previous failures. Note that these failures happened on ub20x64rt1-1 again.

cjjdespres commented 2 years ago

From job/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal/271/, failure in cmdLineTester_fieldwatchtests_0, with output:

Testing: fw001-5
Test start time: 2022/02/24 02:15:55 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer  -Xdump  -agentlib:jvmtitest=test:fw001 -XX:+JITInlineWatches -cp "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 45897 milliseconds
Test result: FAILED
Output from test:

 [OUT] *ERROR*: Invalid number of field reports detected: 0. Expected: 5
 [OUT] 
 [OUT] 
 [OUT] ===============Disabling watches for instanceDoubleField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to remove field watches for instanceDoubleField
 [OUT] 
 [OUT] ===============adding field read watches for instanceIntField
 [OUT] 
 [OUT] 
 [OUT] *ERROR*: Unable to add field watches for instanceIntField

 [OUT] =============== Ending - STATIC AND INSTANCE FIELD WATCH TESTS
 [OUT] 
 [OUT] 
 [OUT] *** Test took 45609 milliseconds
 [OUT] FAILED
 [OUT] 
 [OUT] ield Modification Report #5: method: 0x79dd98160de8 location: 22, fieldID: 0x79dd98160e88, signature: F
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceSingleField. The fieldID is: 0x79dd98160e88 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceSingleField, signature: F, isStatic: No, modificationType: remove
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: add
 [OUT] Field Modification Report #1: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #2: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #3: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #4: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] Field Modification Report #5: method: 0x79dd98160de8 location: 15, fieldID: 0x79dd98160eb0, signature: J
 [OUT] **NOTE**: Modified fieldwatch(, modification) for instanceLongField. The fieldID is: 0x79dd98160eb0 and it is associated with class: com/ibm/jvmti/tests/fieldwatch/MyObject, field: instanceLongField, signature: J, isStatic: No, modificationType: remove
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR]  at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR]  at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR]  at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR]  at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR]  at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR]  at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR]  at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR]  at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR]  at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR]  at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR]  at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR]  at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
 [ERR] Expected: 5, but got: 0
 [ERR] java.lang.Exception: Test failure due to unexpected value.
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.assertEquals(fw001.java:76)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.runTest(fw001.java:158)
 [ERR]  at com.ibm.jvmti.tests.fieldwatch.fw001.testFieldWatch(fw001.java:115)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERR]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERR]  at java.lang.reflect.Method.invoke(Method.java:498)
 [ERR]  at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
 [ERR]  at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
 [ERR]  at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
>> Success condition was not found: [Return code: 0]

There is more in the log that may be relevant, including more error lines, but I hope that is representative.

Version information:

04:43:16  openjdk version "1.8.0_332-internal"
04:43:16  OpenJDK Runtime Environment (build 1.8.0_332-internal-jenkins_2022_02_24_00_02-b00)
04:43:16  Eclipse OpenJ9 VM (build master-016ce8a1e28, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20220224_925 (JIT enabled, AOT enabled)
04:43:16  OpenJ9   - 016ce8a1e28
04:43:16  OMR      - bb570045546
04:43:16  JCL      - bd2e6cf7984 based on jdk8u332-b02)

Reproduced with a 1/50 failure rate with JITAAS in view/Test_grinder/job/Grinder/21338/, and not reproduced with a 100x job in view/Test_grinder/job/Grinder/21375/ without JITAAS.

cjjdespres commented 2 years ago

Very large number of test failures over the past few days. It will take a little while to sort through it all.

cjjdespres commented 2 years ago

I've noticed that there's a large overlap between the failing tests and the ones in this comment, but that may just be coincidence?

Edit: Even though there is such a large overlap (including tests that I haven't had a chance to look at yet), the ones in that comment all failed due to the disabling of SecurityManager, and so it's not directly related.

cjjdespres commented 2 years ago

Two jobs had tests that all failed in the same way - the processes could not connect to a VM, as Monitored VM not ready, then eventually gave up and failed. These failures occurred in:

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/276/

job/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal/275/

Representative log output:

STF 04:44:26.517 - +------ Step 5 - Wait for the processes to complete
STF 04:44:26.517 - | Wait for processes to meet expectations
STF 04:44:26.517 - |   Processes: [LT1, CL1]
STF 04:44:26.517 - |
STF 04:44:26.517 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/27 04:44:26.692 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/27 04:44:26.728 Attempting to connect
CL1 j> 2022/02/27 04:44:26.910 Monitored VM not ready at Feb 27, 2022 4:44:26 AM (attempt 1, elapsed 163ms).
CL1 j> 2022/02/27 04:44:26.910 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:31.912 Attempting to connect
CL1 j> 2022/02/27 04:44:33.498 Monitored VM not ready at Feb 27, 2022 4:44:33 AM (attempt 2, elapsed 6769ms).
CL1 j> 2022/02/27 04:44:33.498 Waiting 5 secs and trying again...
CL1 j> 2022/02/27 04:44:38.499 Attempting to connect
CL1 j> 2022/02/27 04:44:38.587 Monitored VM not ready at Feb 27, 2022 4:44:38 AM (attempt 3, elapsed 11858ms).

CL1 j> 2022/02/27 04:46:55.214 Failed to connect to Monitored VM after 30 attempts in 148 seconds - giving up.  Connection Exception received is below:
CL1 stderr java.rmi.ConnectException: Connection refused to host: 9.24.248.182; nested exception is: 
CL1 stderr  java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr  at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
CL1 stderr  at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
CL1 stderr  at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
CL1 stderr  at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
CL1 stderr  at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
CL1 stderr  at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
CL1 stderr  at com.sun.proxy.$Proxy0.newClient(Unknown Source)
CL1 stderr  at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
CL1 stderr  at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
CL1 stderr  at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ServerConnector.doConnect(ServerConnector.java:273)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ServerConnector.getSecureServerConnection(ServerConnector.java:170)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ServerConnector.<init>(ServerConnector.java:82)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ClassProfiler.<init>(ClassProfiler.java:72)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:89)
CL1 stderr Caused by: java.net.ConnectException: Connection refused (Connection refused)
CL1 stderr  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
CL1 stderr  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
CL1 stderr  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
CL1 stderr  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
CL1 stderr  at java.net.Socket.connect(Socket.java:607)
CL1 stderr  at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288)
CL1 stderr  at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:147)
CL1 stderr  at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
CL1 stderr  at javax.rmi.ssl.SslRMIClientSocketFactory.createSocket(SslRMIClientSocketFactory.java:121)
CL1 stderr  at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
CL1 stderr  ... 14 more
CL1 stderr Exception in thread "main" java.lang.NullPointerException
CL1 stderr  at com.ibm.java.lang.management.internal.ManagementUtils.isANotificationEmitter(ManagementUtils.java:536)
CL1 stderr  at java.lang.management.ManagementFactory.newPlatformMXBeanProxy(ManagementFactory.java:289)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ClassProfiler.getStatsViaProxy(ClassProfiler.java:117)
CL1 stderr  at net.adoptopenjdk.test.jlm.remote.ClassProfiler.main(ClassProfiler.java:102)
STF 04:46:55.496 - **FAILED** Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 - Monitoring Report Summary:
STF 04:46:55.496 -   o Process CL1 ended with exit code (1) and not the expected exit code/s (0)
STF 04:46:55.496 -   o Process LT1 is still running as expected
STF 04:46:55.496 - Killing processes: CL1 LT1
STF 04:46:55.496 -   o Process LT1 pid 2478 stop()
STF 04:47:05.549 -   o Process LT1 pid 2478 terminate()
STF 04:47:06.550 -   o Process LT1 pid 2478 killed
STF 04:47:06.550 -   o Process CL1 pid 2479 is not running
**FAILED** at step 5 (Wait for the processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk8_j9_sanity.system_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../TKG/output_16459519958390/TestJlmRemoteClassAuth_1/20220227-044425-TestJlmRemoteClassAuth/execute.pl line 178.
STF 04:47:06.792 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal/191/, failures in

These all fail in similar ways - the tests attempt to connect to a particular ServerURL, and either just before or just after that a child process fails unexpectedly and the test is aborted.

Representative console log:

STF 02:00:23.344 - +------ Step 3 - Wait for processes to complete
STF 02:00:23.344 - | Wait for processes to meet expectations
STF 02:00:23.344 - |   Processes: [LT1, CL1]
STF 02:00:23.344 - |
STF 02:00:23.345 - Monitoring processes: CL1 LT1
CL1 j> 2022/02/26 02:00:25.833 ServerURL=service:jmx:rmi:///jndi/rmi://localhost:1234/jmxrmi
CL1 j> 2022/02/26 02:00:26.710 Attempting to connect
STF 02:00:28.587 - **FAILED** Process LT1 has ended unexpectedly
STF 02:00:28.587 - Monitoring Report Summary:
STF 02:00:28.587 -   o Process CL1 is still running
STF 02:00:28.587 -   o Process LT1 has ended unexpectedly
STF 02:00:28.588 - Killing processes: CL1 LT1
STF 02:00:28.588 -   o Process LT1 pid 26108 is not running
STF 02:00:28.588 -   o Process CL1 pid 26109 stop()
STF 02:00:38.682 -   o Process CL1 pid 26109 terminate()
STF 02:00:39.700 -   o Process CL1 pid 26109 killed
CL1 j> 2022/02/26 02:00:29.574 Connection established!
CL1 j> 2022/02/26 02:00:30.503 Starting to write data
**FAILED** at step 3 (Wait for processes to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_1645867773491/TestJlmRemoteMemoryNoAuth_0/20220226-020012-TestJlmRemoteMemoryNoAuth/execute.pl line 163.
STF 02:00:39.773 - **FAILED** execute script failed. Expected return value=0 Actual=1

Haven't attempted to reproduce.

pshipton commented 2 years ago

I've noticed that there's a large overlap between the failing tests and the ones in https://github.com/eclipse-openj9/openj9/issues/14412#issuecomment-1027354827, but that may just be coincidence?

If running Java 18+ you need the following set -Djava.security.manager=allow. We do that for the regular tests by setting the JAVA_TOOL_OPTIONS environment variable to prepend the option https://github.com/adoptium/aqa-tests/pull/3352

cjjdespres commented 2 years ago

The next jobs all fail due to a segfault. They also have some tests that seem to fail for other reasons, so I'll post those separately. The two running the functional tests all have identical stack backtraces

----------- Stack Backtrace -----------
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x133da (0x00007F0F643EF1BA [libj9vm29.so+0xa61ba])
bytecodeLoopCompressed+0x95 (0x00007F0F643DBDD5 [libj9vm29.so+0x92dd5])
 (0x00007F0F6447F5B2 [libj9vm29.so+0x1365b2])

aside from what I assume are the memory addresses. The one running the system tests has more varied stack backtraces. The jobs and tests are:

job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/

job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/

job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/

Haven't been able to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/ (also discussed above), failures in

They all seem to fail in the same way, but some have less detail in their errors. All but the ContendedFieldsTests and the testDDRExtJunit tests fail with messages like:

TestAttachErrorHandling_0 Start Time: Mon Feb 28 00:49:39 2022 Epoch Time (ms): 1646038179205
variation: NoOptions
JVM_OPTIONS: -XX:+UseJITServer 
Failed to register natives for jdk.internal.reflect.ConstantPool
Exception in thread "main" java/lang/ExceptionInInitializerError
    at java/lang/J9VMInternals.ensureError (java.base@17.0.3-internal/J9VMInternals.java:185)
    at java/lang/J9VMInternals.recordInitializationFailure (java.base@17.0.3-internal/J9VMInternals.java:174)
java/lang/NullPointerException
    at com/ibm/jit/JITHelpers.getClassDepthAndFlagsFromJ9Class64 (java.base@17.0.3-internal/NativeMethod:4294967295)
    at com/ibm/jit/JITHelpers.isArray (java.base@17.0.3-internal/JITHelpers.java:141)
    at java/lang/J9VMInternals.fastIdentityHashCode (java.base@17.0.3-internal/J9VMInternals.java:408)
    at java/lang/Object.hashCode (java.base@17.0.3-internal/Object.java:120)
    at java/util/HashMap.hash (java.base@17.0.3-internal/HashMap.java:338)
    at java/util/HashMap.getNode (java.base@17.0.3-internal/HashMap.java:568)
    at java/util/HashMap.get (java.base@17.0.3-internal/HashMap.java:556)
    at jdk/internal/reflect/Reflection.registerFilter (java.base@17.0.3-internal/Reflection.java:286)
    at jdk/internal/reflect/Reflection.registerFieldsToFilter (java.base@17.0.3-internal/Reflection.java:273)
    at jdk/internal/reflect/ConstantPool.<clinit> (java.base@17.0.3-internal/ConstantPool.java:108)

TestAttachErrorHandling_0_FAILED

The testDDREExtJunit tests have their TAP results cut off, but in the console log they have that same error but without the "failed to register natives" line. The ContendedFieldTests have only a NullPointerException reported.

Haven't been able to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_extended.functional_x86-64_linux_jit_Personal/201/, also discussed above, failure in

Console log:

Exception in thread "main" Exception in thread "Attach API initializer" java.lang.InternalError: not invoker: invokeExactjava.lang.InternalError: not invoker: invokeExact

    at java.base/java.lang.invoke.Invokers.methodHandleInvokeLinkerMethod(Invokers.java:260)    at java.base/java.lang.invoke.Invokers.methodHandleInvokeLinkerMethod(Invokers.java:260)

    at java.base/java.lang.invoke.MethodHandleNatives.linkMethodImpl(MethodHandleNatives.java:482)
    at java.base/java.lang.invoke.MethodHandleNatives.linkMethod(MethodHandleNatives.java:473)
    at java.base/java.lang.invoke.MethodHandleNatives.linkMethodImpl(MethodHandleNatives.java:482)  at java.base/java.lang.invoke.MethodHandleResolver.linkCallerMethod(MethodHandleResolver.java:457)

    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
    at java.base/java.lang.invoke.MethodHandleNatives.linkMethod(MethodHandleNatives.java:473)  at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
    at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:196)

    at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:467)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:488)
    at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:174)
    at java.base/java.lang.invoke.MethodHandleResolver.linkCallerMethod(MethodHandleResolver.java:457)  at java.base/java.lang.Module.getResourceAsStream(Module.java:1671)

    at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
    at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:683)
    at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:874)
    at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:868)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)  at java.base/java.lang.Thread.uncaughtException(Thread.java:1403)

    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
    at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:196)
    at java.base/java.util.ResourceBundle.getLoader(ResourceBundle.java:584)
    at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1574)
    at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1549)
    at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:858)
    at java.base/sun.launcher.LauncherHelper$ResourceBundleHolder.<clinit>(LauncherHelper.java:125)
    at java.base/sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:483)
    at java.base/sun.launcher.LauncherHelper.abort(LauncherHelper.java:647)
    at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:807)
    at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:686)

StackWalkerTest_1_FAILED

Haven't been able to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_extended.system_ppc64le_linux_jit_Personal/193/, also discussed above, failure in:

Excerpt from console log:

MLT stderr Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.logging.log4j.message.DefaultFlowMessageFactory
MLT stderr  at org.apache.logging.log4j.spi.AbstractLogger.<clinit>(AbstractLogger.java:101)
MLT stderr  at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:61)
MLT stderr  at net.adoptopenjdk.loadTest.LoadTest.<clinit>(LoadTest.java:46)
MLT stderr Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.DefaultFlowMessageFactory
MLT stderr  at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:827)
MLT stderr  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
MLT stderr  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1091)
MLT stderr  ... 3 more
STF 01:41:55.438 - **FAILED** Process MLT ended with exit code (1) and not the expected exit code/s (0)

This seems more like a misconfigured system or a failure in the test itself, since the test can't find some component of log4j.

Haven't been able to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/278/, also discussed above, failure in:

Excerpt from console log:

===============================================
Running test cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2 ...
===============================================
cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2 Start Time: Mon Feb 28 01:30:51 2022 Epoch Time (ms): 1646040651529
variation: Mode607-OSRG
JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation 
Error: Could not find or load main class MainTester

cmdLineTester_jvmtitests_hcr_OSRG_nongold_SE80_2_FAILED

This also seems like a failure in the test itself.

Haven't been able to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/274/, failure in:

Console log excerpt:

04:40:57  STF 01:40:57.545 - Monitoring processes: LT 
04:40:58  LT  stderr Exception in thread "main" java/lang/NoClassDefFoundError: java.ut����.����������������.����������������
04:40:58  LT  stderr    at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString (java.base@11.0.15-internal/MethodTypeHelper.java:259)
04:40:58  LT  stderr    at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic (java.base@11.0.15-internal/MethodHandleResolver.java:203)
04:40:58  LT  stderr    at jdk/internal/module/ModuleBootstrap.decode (java.base@11.0.15-internal/ModuleBootstrap.java:882)
04:40:58  LT  stderr    at jdk/internal/module/ModuleBootstrap.decode (java.base@11.0.15-internal/ModuleBootstrap.java:905)
04:40:58  LT  stderr    at jdk/internal/module/ModuleBootstrap.addExtraExportsAndOpens (java.base@11.0.15-internal/ModuleBootstrap.java:674)
04:40:58  LT  stderr    at jdk/internal/module/ModuleBootstrap.boot (java.base@11.0.15-internal/ModuleBootstrap.java:432)
04:40:58  LT  stderr    at java/lang/ClassLoader.initializeClassLoaders (java.base@11.0.15-internal/ClassLoader.java:211)
04:40:58  LT  stderr    at java/lang/Thread.initialize (java.base@11.0.15-internal/Thread.java:462)
04:40:58  LT  stderr    at java/lang/Thread.<init> (java.base@11.0.15-internal/Thread.java:162)
04:40:58  STF 01:40:57.816 - **FAILED** Process LT  ended with exit code (1) and not the expected exit code/s (0)
04:40:58  STF 01:40:57.817 - Monitoring Report Summary:
04:40:58  STF 01:40:57.817 -   o Process LT  ended with exit code (1) and not the expected exit code/s (0)
04:40:58  STF 01:40:57.817 - Killing processes: LT 
04:40:58  STF 01:40:57.817 -   o Process LT  pid 2374985 is not running
04:40:58  **FAILED** at step 1 (Run mixed unit tests). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal_testList_2/aqa-tests/TKG/../TKG/output_16460406309573/MiniMix_aot_5m_0/20220228-014051-MixedLoadTest/execute.pl line 98.
04:40:58  STF 01:40:57.826 - **FAILED** execute script failed. Expected return value=0 Actual=1

The TAP extended test results are cut off partway through the first line of the error; this is from the full console log.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/894/, failures in:

The shrtest_linux_0 test fails with output:

----------------------------------------
CacheDirPerm Test
----------------------------------------

CacheDirPerm test begin...

    INFO: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(487) testCacheDirPerm Test 0: cacheType: persistent, useExistingDir: 1, cacheDirPermStr: <NULL>, expectedDirPerm: 496, useDefaultDir: 0

    INFO: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(494) testCacheDirPerm Initial cleanup

    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(240) removeTempDir Failed to unlink /tmp/testParentDir//testCacheDirPermDir/javasharedresources

    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(240) removeTempDir Failed to unlink /tmp/testParentDir//testCacheDirPermDir

    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(240) removeTempDir Failed to unlink /tmp/testParentDir/

    INFO: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(509) testCacheDirPerm Created cache dir /tmp/testParentDir/testCacheDirPermDir/

    INFO: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/OpenCacheHelper.cpp(50) openTestCache Opening test cache
    INFO: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/OpenCacheHelper.cpp(235) openTestCache Done opening test cache
    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(534) testCacheDirPerm Incorrect permission on cache directory. Expected: 496, found: 1023

    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(240) removeTempDir Failed to unlink /tmp/testParentDir/testCacheDirPermDir//javasharedresources

    ERROR: /home/jenkins/workspace/Build_JDK11_x86-64_linux_jit_Personal/openj9/runtime/tests/shared/CacheDirPerm.cpp(240) removeTempDir Failed to unlink /tmp/testParentDir/testCacheDirPermDir/

Result of test CacheDirPerm: FAILED

which looks like a system error.

The J9vmTest_4 fails with output

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2022/02/26 11:33:41 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/core.20220226.113341.21804.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/core.20220226.113341.21804.0001.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/heapdump.20220226.113341.21804.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/heapdump.20220226.113341.21804.0002.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/javacore.20220226.113341.21804.0003.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/javacore.20220226.113341.21804.0003.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/Snap.20220226.113341.21804.0004.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal_testList_1/aqa-tests/TKG/output_16458654308257/J9vmTest_4/Snap.20220226.113341.21804.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at j9vm.test.arraylets.ArrayletAllocateTest.main(ArrayletAllocateTest.java:79)
no-zero exit value: 1
*** Test FAILED *** (j9vm.test.arraylets.ArrayletAllocateTest)

which also looks like a system error.

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal/195/, failure in:

Console log excerpt:

LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x00020002
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
LT  stderr Handler1=00007F686D978140 Handler2=00007F686D6D5190 InaccessibleAddress=0000000000000000
LT  stderr RDI=00007F686805F1D0 RSI=0000000000000007 RAX=0000000000000000 RBX=00007F6828456DD0
LT  stderr RCX=0000000000000000 RDX=0000000099669966 R8=00007F67D4B506B0 R9=00007F67D4C4F1A0
LT  stderr R10=0000000000000006 R11=000000000000000A R12=00007F6866BAF360 R13=00007F6868AD15A8
LT  stderr R14=00007F67D4C4EFE0 R15=0000000000000000
LT  stderr RIP=00007F6866884DF9 GS=0000 FS=0000 RSP=00007F67D4C4EFD0
LT  stderr EFlags=0000000000010246 CS=0033 RBP=00007F67D4C4F520 ERR=0000000000000004
LT  stderr TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
LT  stderr xmm0 0000003000000020 (f: 32.000000, d: 1.018558e-312)
LT  stderr xmm1 2d3638785f37314b (f: 1597452672.000000, d: 6.817684e-91)
LT  stderr xmm2 7361622f63672f72 (f: 1667706752.000000, d: 6.077214e+247)
LT  stderr xmm3 2d3638785f37314b (f: 1597452672.000000, d: 6.817684e-91)
LT  stderr xmm4 3fd4043057b6ee09 (f: 1471606272.000000, d: 3.127557e-01)
LT  stderr xmm5 006e00650047002e (f: 4653102.000000, d: 1.335113e-306)
LT  stderr xmm6 00650074006e0069 (f: 7209065.000000, d: 9.346098e-307)
LT  stderr xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm8 6332313578766100 (f: 2021024000.000000, d: 6.865676e+169)
LT  stderr xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
LT  stderr Module_base_address=00007F68666E4000
LT  stderr Target=2_90_20220224_204 (Linux 4.15.0-167-generic)
LT  stderr CPU=amd64 (4 logical CPUs) (0xf67fe000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr _ZN26MM_MarkingSchemeRootMarker11doStackSlotEPP8J9ObjectPvPKv+0x389 (0x00007F6866884DF9 [libj9gc_full29.so+0x1a0df9])
LT  stderr jitWalkFrame+0x16b (0x00007F6867501F6B [libj9jit29.so+0x950f6b])
LT  stderr jitWalkStackFrames+0xddc (0x00007F686750335C [libj9jit29.so+0x95235c])
LT  stderr walkStackFrames+0xb3 (0x00007F686D9B8113 [libj9vm29.so+0x80113])
LT  stderr _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x76 (0x00007F6866725F26 [libj9gc_full29.so+0x41f26])
LT  stderr _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0xfd (0x00007F686671E40D [libj9gc_full29.so+0x3a40d])
LT  stderr _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xbf (0x00007F686671D14F [libj9gc_full29.so+0x3914f])
LT  stderr _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0x42 (0x00007F686671FCA2 [libj9gc_full29.so+0x3bca2])
LT  stderr _ZN18MM_MarkingDelegate9scanRootsEP18MM_EnvironmentBaseb+0x14b (0x00007F686687FA0B [libj9gc_full29.so+0x19ba0b])
LT  stderr _ZN19MM_ParallelMarkTask3runEP18MM_EnvironmentBase+0x79 (0x00007F68668A47B9 [libj9gc_full29.so+0x1c07b9])
LT  stderr _ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0x9e (0x00007F68668120DE [libj9gc_full29.so+0x12e0de])
LT  stderr _ZN19MM_ParallelGlobalGC7markAllEP18MM_EnvironmentBaseb+0x135 (0x00007F686682CF85 [libj9gc_full29.so+0x148f85])
LT  stderr _ZN19MM_ParallelGlobalGC24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x1a0 (0x00007F686682D950 [libj9gc_full29.so+0x149950])
LT  stderr _ZN15MM_ConcurrentGC22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0x26 (0x00007F68668AE616 [libj9gc_full29.so+0x1ca616])
LT  stderr _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0xb6 (0x00007F68667EF9D6 [libj9gc_full29.so+0x10b9d6])
LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x124 (0x00007F686680BD84 [libj9gc_full29.so+0x127d84])
LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x2c (0x00007F686680BC8C [libj9gc_full29.so+0x127c8c])
LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x2c (0x00007F686680BC8C [libj9gc_full29.so+0x127c8c])
LT  stderr j9gc_modron_global_collect_with_overrides+0x38 (0x00007F68667121F8 [libj9gc_full29.so+0x2e1f8])
LT  stderr JVM_GC_Impl+0x6a (0x00007F6865D10F7A [libjclse29.so+0x61f7a])
LT  stderr  (0x00007F680E47C46C [<unknown>+0x0])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/02/25 03:26:17 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/core.20220225.032617.31678.0001.dmp' in response to an event
LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/core.20220225.032617.31678.0001.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/javacore.20220225.032617.31678.0002.txt' in response to an event
STF 03:26:20.658 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/javacore.20220225.032617.31678.0002.txt
STF 03:26:20.659 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/core.20220225.032617.31678.0001.dmp
LT  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/javacore.20220225.032617.31678.0002.txt
LT  stderr core file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/core.20220225.032617.31678.0001.dmp
LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/javacore.20220225.032617.31678.0002.txt
LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/Snap.20220225.032617.31678.0003.trc' in response to an event
STF 03:26:22.162 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/Snap.20220225.032617.31678.0003.trc
LT  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/Snap.20220225.032617.31678.0003.trc
LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/Snap.20220225.032617.31678.0003.trc
LT  stderr JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_jit_Personal_testList_3/aqa-tests/TKG/output_1645778386787/LambdaLoadTest_special_J9_5m_9/20220225-032505-LambdaLoadTest/results/jitdump.20220225.032617.31678.0004.dmp' in response to an event
LT  stderr JVMDUMP051I JIT dump occurred in 'load-1' thread 0x00007F6868ACF800
LT  stderr JVMDUMP053I JIT dump is recompiling net/adoptopenjdk/test/lambda/TestLambdaCapture.testCaptureArrayList()V

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/269/, various failures. I haven't been able to reproduce them yet. I've broken them down by cause here:

Segfaults

Console log for DaaLoadTest_daa1_special_5m_17:

DLT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00000000
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00007FFF97474C30 Handler2=00007FFF9739C900
DLT stderr R0=0000000000000000 R1=00007FFF97B9B220 R2=00007FFF97686E00 R3=0000000000179578
DLT stderr R4=0000000000179590 R5=0000000000179518 R6=0000000000000003 R7=0000000000000004
DLT stderr R8=0000000000000000 R9=00007FFF39464B70 R10=0000000000000000 R11=0000000000179588
DLT stderr R12=00007FFF97CA22A0 R13=00007FFF97BA68F0 R14=0000000000179570 R15=0000000000095700
DLT stderr R16=00007FFF719A8038 R17=FFFFFFFFFFFFFFFF R18=0000000000000001 R19=00007FFF3947F0BD
DLT stderr R20=0000000000000008 R21=0000000000000000 R22=0000000000179540 R23=0000000000000001
DLT stderr R24=0000000000095700 R25=00007FFF901D0301 R26=00007FFF96C3F100 R27=00007FFF962BC4F0
DLT stderr R28=0000000000000002 R29=0000000000000002 R30=FFFFFFFFFFFFFFFD R31=00000000852ADD58
DLT stderr NIP=00007FFF974FC548 MSR=800000000280F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF974F0218
DLT stderr LINK=00007FFF974E6FF4 XER=0000000000000000 CCR=0000000022242242 SOFTE=0000000000000001
DLT stderr TRAP=0000000000000300 DAR=00007FFFBE7128C8 dsisr=0000000040000000 RESULT=0000000000000000
DLT stderr FPR0 00000000001795d8 (f: 1545688.000000, d: 7.636713e-318)
DLT stderr FPR1 4051732fc0000000 (f: 3221225472.000000, d: 6.979979e+01)
DLT stderr FPR2 41e0000000000000 (f: 0.000000, d: 2.147484e+09)
DLT stderr FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
DLT stderr FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
DLT stderr FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
DLT stderr FPR7 3fd1058bc8a07ee1 (f: 3365961472.000000, d: 2.659635e-01)
DLT stderr FPR8 bfdffffef20a4123 (f: 4060758272.000000, d: -4.999997e-01)
DLT stderr FPR9 bfd00ea348b88334 (f: 1220051712.000000, d: -2.508934e-01)
DLT stderr FPR10 bfe02cfb046d5655 (f: 74274392.000000, d: -5.054908e-01)
DLT stderr FPR11 3f31c55b43f8e679 (f: 1140385408.000000, d: 2.711628e-04)
DLT stderr FPR12 3fc7565060000000 (f: 1610612736.000000, d: 1.823216e-01)
DLT stderr FPR13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
DLT stderr FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal_testList_4/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
DLT stderr Module_base_address=00007FFF97430000
DLT stderr Target=2_90_20220228_951 (Linux 5.3.18-150300.59.49-default)
DLT stderr CPU=ppc64le (4 logical CPUs) (0x1ed4c0000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr bytecodeLoopCompressed+0x162b8 (0x00007FFF974FC548 [libj9vm29.so+0xcc548])
DLT stderr  (0x00007FFF975A62A0 [libj9vm29.so+0x1762a0])
DLT stderr sendClinit+0x1fc (0x00007FFF97446D1C [libj9vm29.so+0x16d1c])
DLT stderr initializeImpl+0x3bc (0x00007FFF9745172C [libj9vm29.so+0x2172c])
DLT stderr classInitStateMachine+0x878 (0x00007FFF97452308 [libj9vm29.so+0x22308])
DLT stderr findClass+0x110 (0x00007FFF9748A3D0 [libj9vm29.so+0x5a3d0])
DLT stderr gpCheckFindClass+0x3c (0x00007FFF9747B67C [libj9vm29.so+0x4b67c])
DLT stderr initializeSunReflectConstantPoolIDCache+0xe0 (0x00007FFF95770520 [libjclse29.so+0x30520])
DLT stderr registerJdkInternalReflectConstantPoolNatives+0xd8 (0x00007FFF957A6298 [libjclse29.so+0x66298])
DLT stderr standardInit+0x14a8 (0x00007FFF95796AE8 [libjclse29.so+0x56ae8])
DLT stderr scarInit+0x10c (0x00007FFF957AA7BC [libjclse29.so+0x6a7bc])
DLT stderr J9VMDllMain+0x378 (0x00007FFF957AB558 [libjclse29.so+0x6b558])
DLT stderr runJ9VMDllMain+0x120 (0x00007FFF97491B30 [libj9vm29.so+0x61b30])
DLT stderr pool_do+0xc4 (0x00007FFF975CAF74 [libj9vm29.so+0x19af74])
DLT stderr runInitializationStage+0x108 (0x00007FFF974416C8 [libj9vm29.so+0x116c8])
DLT stderr protectedInitializeJavaVM+0x2dd0 (0x00007FFF974A1E20 [libj9vm29.so+0x71e20])
DLT stderr omrsig_protect+0x358 (0x00007FFF9739DD08 [libj9prt29.so+0x3dd08])
DLT stderr initializeJavaVM+0x560 (0x00007FFF97498790 [libj9vm29.so+0x68790])
DLT stderr J9_CreateJavaVM+0xa4 (0x00007FFF97485D64 [libj9vm29.so+0x55d64])
DLT stderr JNI_CreateJavaVM_impl+0xd54 (0x00007FFF97ACBCA4 [libjvm.so+0x1bca4])
DLT stderr JNI_CreateJavaVM+0xc88 (0x00007FFF97BB2898 [libjvm.so+0x12898])
DLT stderr JavaMain+0xd0 (0x00007FFF97E3B150 [libjli.so+0xb150])
DLT stderr ThreadJavaMain+0x18 (0x00007FFF97E42128 [libjli.so+0x12128])
DLT stderr start_thread+0xf8 (0x00007FFF97E796A8 [libpthread.so.0+0x96a8])
DLT stderr clone+0x74 (0x00007FFF97D17CD4 [libc.so.6+0x137cd4])

Null property: java.home

Console log for DaaLoadTest_daa1_special_5m_18:

DLT stderr Exception in thread "main" java/lang/InternalError: null property: java.home
DLT stderr  at jdk/internal/util/StaticProperty.initProperty (java.base@11/StaticProperty.java:52)
DLT stderr  at jdk/internal/util/StaticProperty.<clinit> (java.base@11/StaticProperty.java:41)
DLT stderr  at java/lang/System.ensureProperties (java.base@11/System.java:443)
DLT stderr  at java/lang/System.afterClinitInitialization (java.base@11/System.java:201)
DLT stderr  at java/lang/Thread.initialize (java.base@11/Thread.java:410)
DLT stderr  at java/lang/Thread.<init> (java.base@11/Thread.java:162)

Path/URI errors

All of these have illegal character exceptions, or string index out of bound exceptions, in some way.

Console log for DaaLoadTest_daa2_special_5m_22

DLT stderr Exception in thread "Attach API initializer" java.lang.NoClassDefFoundError: com.ibm.oti.util.Msg (initialization failure)
DLT stderr  at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:135)
DLT stderr  at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:865)
DLT stderr  at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:859)
DLT stderr  at java.base/java.lang.Thread.uncaughtException(Thread.java:1385)
DLT stderr Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 2: file:///
DLT stderr  at java.base/java.net.URI.create(URI.java:883)
DLT stderr  at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:107)
DLT stderr  at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>(FileSystems.java:94)
DLT stderr  at java.base/java.nio.file.FileSystems.getDefault(FileSystems.java:182)
DLT stderr  at java.base/java.nio.file.Path.of(Path.java:147)
DLT stderr  at java.base/java.nio.file.Paths.get(Paths.java:69)
DLT stderr  at java.base/jdk.internal.jimage.ImageReaderFactory.<clinit>(ImageReaderFactory.java:51)
DLT stderr Error: A JNI error has occurred, please check your installation and try again
DLT stderr Exception in thread "main"   at java.base/jdk.internal.module.SystemModuleFinders$SystemImage.<clinit>(SystemModuleFinders.java:383)
DLT stderr java.lang.BootstrapMethodError: java.lang.StringIndexOutOfBoundsException
DLT stderr  at java.base/java.util.ResourceBundle.getLoader(ResourceBundle.java:583)
DLT stderr  at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1581)
DLT stderr  at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1556)
DLT stderr  at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.findImageLocation(SystemModuleFinders.java:426) at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:857)
DLT stderr 
DLT stderr  at java.base/sun.launcher.LauncherHelper$ResourceBundleHolder.<clinit>(LauncherHelper.java:124)
DLT stderr  at java.base/sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:452)
DLT stderr  at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.read(SystemModuleFinders.java:464)  at java.base/sun.launcher.LauncherHelper.abort(LauncherHelper.java:616)
DLT stderr  at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:781)
DLT stderr 
DLT stderr  at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:655)
DLT stderr Caused by: java.lang.StringIndexOutOfBoundsException
DLT stderr  at java.base/java.lang.String.charAt(String.java:1327)
DLT stderr  at java.base/jdk.internal.org.objectweb.asm.Type.getArgumentsAndReturnSizes(Type.java:455)
DLT stderr  at java.base/jdk.internal.org.objectweb.asm.MethodWriter.<init>(MethodWriter.java:514)
DLT stderr  at java.base/jdk.internal.org.objectweb.asm.ClassWriter.visitMethod(ClassWriter.java:856)   at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:448)
DLT stderr  at java.base/java.lang.invoke.InnerClassLambdaMetafactory.generateConstructor(InnerClassLambdaMetafactory.java:359)
DLT stderr 
DLT stderr  at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:284)
DLT stderr  at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:434)   at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:195)
DLT stderr  at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
DLT stderr 
DLT stderr  at java.base/java.lang.invoke.MethodHandleResolver.invokeBsm(MethodHandleResolver.java:182)
DLT stderr  at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:238)
DLT stderr  ... 9 more  at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:146)
DLT stderr 
DLT stderr  at java.base/java.lang.Module.getResourceAsStream(Module.java:1590)
DLT stderr  at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
DLT stderr  at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
DLT stderr  at java.base/java.security.AccessController.doPrivileged(AccessController.java:682)
DLT stderr  at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
DLT stderr  at java.base/java.lang.J9VMInternals.formatNoSuchMethod(J9VMInternals.java:521)
DLT stderr  at java.base/java.lang.invoke.PrimitiveHandle.lookupMethod(Native Method)
DLT stderr  at java.base/java.lang.invoke.PrimitiveHandle.finishMethodInitialization(PrimitiveHandle.java:196)
DLT stderr  at java.base/java.lang.invoke.DirectHandle.<init>(DirectHandle.java:51)
DLT stderr  at java.base/java.lang.invoke.DirectHandle.<init>(DirectHandle.java:43)
DLT stderr  at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:648)
DLT stderr  at java.base/java.lang.invoke.VarHandle.populateMHs(VarHandle.java:949)
DLT stderr  at java.base/java.lang.invoke.InstanceFieldVarHandle.populateMHs(InstanceFieldVarHandle.java:83)
DLT stderr  at java.base/java.lang.invoke.InstanceFieldVarHandle.<init>(InstanceFieldVarHandle.java:95)
DLT stderr  at java.base/java.lang.invoke.MethodHandles$Lookup.findVarHandleCommon(MethodHandles.java:1567)
DLT stderr  at java.base/java.lang.invoke.MethodHandles$Lookup.findVarHandle(MethodHandles.java:1540)
DLT stderr  at java.base/java.util.concurrent.atomic.AtomicBoolean.<clinit>(AtomicBoolean.java:57)
DLT stderr  at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:74)
DLT stderr  at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
DLT stderr  at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:127)
DLT stderr  at java.base/openj9.internal.tools.attach.target.FileLock.lockFile(FileLock.java:106)
DLT stderr  at java.base/openj9.internal.tools.attach.target.CommonDirectory.obtainControllerLock(CommonDirectory.java:148)
DLT stderr  at java.base/openj9.internal.tools.attach.target.AttachHandler.createFiles(AttachHandler.java:190)
DLT stderr  at java.base/openj9.internal.tools.attach.target.AttachHandler.initialize(AttachHandler.java:298)
DLT stderr  at java.base/openj9.internal.tools.attach.target.AttachHandler.run(AttachHandler.java:231)
DLT stderr Caused by: java.net.URISyntaxException: Illegal character in path at index 2: file:///
DLT stderr  at java.base/java.net.URI$Parser.fail(URI.java:2913)
DLT stderr  at java.base/java.net.URI$Parser.checkChars(URI.java:3084)
DLT stderr  at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3166)
DLT stderr  at java.base/java.net.URI$Parser.parse(URI.java:3125)
DLT stderr  at java.base/java.net.URI.<init>(URI.java:600)
DLT stderr  at java.base/java.net.URI.create(URI.java:881)
DLT stderr  ... 37 more
cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/192/ , failure in DaaLoadTest_daa1_special_5m_21. A segfault.

Console log:

STF 01:54:00.148 - Monitoring processes: DLT
DLT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00000000
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00003FFF7B455AB0 Handler2=00003FFF7B37C900
DLT stderr R0=0000000088719ED8 R1=00003FFF7BA59690 R2=00000000000AD0B0 R3=00003FFF19464B50
DLT stderr R4=0000000000000000 R5=000000000000000E R6=FFFFFFFFFFFFFFFF R7=00003FFF527A8038
DLT stderr R8=00000000000952A8 R9=000000000000000E R10=00003FFF52E307E0 R11=00003FFF7A6B791C
DLT stderr R12=0000000040222444 R13=00003FFF7BA66900 R14=00000000000952A0 R15=0000000000095700
DLT stderr R16=00003FFF527A8038 R17=FFFFFFFFFFFFFFFF R18=00003FFF7BA59A10 R19=0000000000000000
DLT stderr R20=0000000000000000 R21=0000000000000000 R22=0000000000000000 R23=0000000000000000
DLT stderr R24=0000000000000000 R25=0000000000000000 R26=0000000000000000 R27=00003FFF7BA59C68
DLT stderr R28=FFFFFFFFFFFFFFFF R29=0000000000000003 R30=0000000000000000 R31=000000000014DE00
DLT stderr NIP=00003FFF52E30820 MSR=800000010280F033 ORIG_GPR3=C0000000000027E0 CTR=00003FFF52E307E8
DLT stderr LINK=00003FFF7A6B6FF4 XER=0000000020000000 CCR=0000000028222444 SOFTE=0000000000000001
DLT stderr TRAP=0000000000000300 DAR=00003FFFA1B7EA28 dsisr=0000000040000000 RESULT=0000000000000000
DLT stderr FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR1 405562c540000000 (f: 1073741824.000000, d: 8.554329e+01)
DLT stderr FPR2 3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
DLT stderr FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR4 3fe3333340000000 (f: 1073741824.000000, d: 6.000000e-01)
DLT stderr FPR5 3fca181e80000000 (f: 2147483648.000000, d: 2.038611e-01)
DLT stderr FPR6 3fe5555560000000 (f: 1610612736.000000, d: 6.666667e-01)
DLT stderr FPR7 3f2b4e8780000000 (f: 2147483648.000000, d: 2.083340e-04)
DLT stderr FPR8 4037000000000000 (f: 0.000000, d: 2.300000e+01)
DLT stderr FPR9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR10 bf859070e0000000 (f: 3758096384.000000, d: -1.052941e-02)
DLT stderr FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
DLT stderr FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR13 3fdb9b2820000000 (f: 536870912.000000, d: 4.313450e-01)
DLT stderr FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr 
DLT stderr Compiled_method=java/lang/J9VMInternals.fastIdentityHashCode(Ljava/lang/Object;)I
DLT stderr Target=2_90_20220228_205 (Linux 3.10.0-1160.53.1.el7.ppc64le)
DLT stderr CPU=ppc64le (4 logical CPUs) (0x1de3d0000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr protectedBacktrace+0x24 (0x00003FFF7B376B04 [libj9prt29.so+0x36b04])
DLT stderr omrsig_protect+0x358 (0x00003FFF7B37DD08 [libj9prt29.so+0x3dd08])
DLT stderr omrintrospect_backtrace_thread_raw+0xe4 (0x00003FFF7B377134 [libj9prt29.so+0x37134])
DLT stderr protectedIntrospectBacktraceThread+0x24 (0x00003FFF7B3767E4 [libj9prt29.so+0x367e4])
DLT stderr omrsig_protect+0x358 (0x00003FFF7B37DD08 [libj9prt29.so+0x3dd08])
DLT stderr omrintrospect_backtrace_thread+0x90 (0x00003FFF7B3768E0 [libj9prt29.so+0x368e0])
DLT stderr generateDiagnosticFiles+0x140 (0x00003FFF7B455470 [libj9vm29.so+0x45470])
DLT stderr omrsig_protect+0x358 (0x00003FFF7B37DD08 [libj9prt29.so+0x3dd08])
DLT stderr structuredSignalHandler+0x1b4 (0x00003FFF7B4557A4 [libj9vm29.so+0x457a4])
DLT stderr structuredSignalHandlerVM+0x50 (0x00003FFF7B455B00 [libj9vm29.so+0x45b00])
DLT stderr mainSynchSignalHandler+0x260 (0x00003FFF7B37CB60 [libj9prt29.so+0x3cb60])
DLT stderr __kernel_sigtramp_rt64+0x0 (0x00003FFF7BD90478)
DLT stderr  (0x00003FFF7ACD1010 [libj9jit29.so+0xbe1010])
DLT stderr sendClinit+0x1fc (0x00003FFF7B4274DC [libj9vm29.so+0x174dc])
DLT stderr initializeImpl+0x3bc (0x00003FFF7B43201C [libj9vm29.so+0x2201c])
DLT stderr classInitStateMachine+0x878 (0x00003FFF7B432C08 [libj9vm29.so+0x22c08])
DLT stderr classInitStateMachine+0x1c8c (0x00003FFF7B43401C [libj9vm29.so+0x2401c])
DLT stderr classInitStateMachine+0x1c8c (0x00003FFF7B43401C [libj9vm29.so+0x2401c])
DLT stderr resolveClassRef+0x310 (0x00003FFF7B49FA40 [libj9vm29.so+0x8fa40])
DLT stderr bytecodeLoopCompressed+0x2188 (0x00003FFF7B4CA198 [libj9vm29.so+0xba198])
DLT stderr  (0x00003FFF7B57F4E8 [libj9vm29.so+0x16f4e8])
DLT stderr sendClinit+0x1fc (0x00003FFF7B4274DC [libj9vm29.so+0x174dc])
DLT stderr initializeImpl+0x3bc (0x00003FFF7B43201C [libj9vm29.so+0x2201c])
DLT stderr classInitStateMachine+0x878 (0x00003FFF7B432C08 [libj9vm29.so+0x22c08])
DLT stderr resolveStaticMethodRefInto+0x464 (0x00003FFF7B4A0354 [libj9vm29.so+0x90354])
DLT stderr resolveStaticMethodRef+0x44 (0x00003FFF7B4A0504 [libj9vm29.so+0x90504])
DLT stderr bytecodeLoopCompressed+0xe234 (0x00003FFF7B4D6244 [libj9vm29.so+0xc6244])
DLT stderr  (0x00003FFF7B57F4E8 [libj9vm29.so+0x16f4e8])
DLT stderr initializeAttachedThreadImpl+0x32c (0x00003FFF7B428B3C [libj9vm29.so+0x18b3c])
DLT stderr initializeAttachedThread+0x60 (0x00003FFF7B4302F0 [libj9vm29.so+0x202f0])
DLT stderr standardInit+0x1098 (0x00003FFF79B07EB8 [libjclse29.so+0x57eb8])
DLT stderr scarInit+0x10c (0x00003FFF79B1B6CC [libjclse29.so+0x6b6cc])
DLT stderr J9VMDllMain+0x378 (0x00003FFF79B1C468 [libjclse29.so+0x6c468])
DLT stderr runJ9VMDllMain+0x120 (0x00003FFF7B472160 [libj9vm29.so+0x62160])
DLT stderr pool_do+0xc4 (0x00003FFF7B5A43F4 [libj9vm29.so+0x1943f4])
DLT stderr runInitializationStage+0x108 (0x00003FFF7B421958 [libj9vm29.so+0x11958])
DLT stderr protectedInitializeJavaVM+0x2de0 (0x00003FFF7B482550 [libj9vm29.so+0x72550])
DLT stderr omrsig_protect+0x358 (0x00003FFF7B37DD08 [libj9prt29.so+0x3dd08])
DLT stderr initializeJavaVM+0x560 (0x00003FFF7B478DE0 [libj9vm29.so+0x68de0])
DLT stderr J9_CreateJavaVM+0xa4 (0x00003FFF7B466084 [libj9vm29.so+0x56084])
DLT stderr JNI_CreateJavaVM_impl+0xd54 (0x00003FFF7B97ED44 [libjvm.so+0x1ed44])
DLT stderr JNI_CreateJavaVM+0xc88 (0x00003FFF7BA72D08 [libjvm.so+0x12d08])
DLT stderr JavaMain+0xd0 (0x00003FFF7BD0BCB0 [libjli.so+0xbcb0])
DLT stderr ThreadJavaMain+0x18 (0x00003FFF7BD11E28 [libjli.so+0x11e28])
DLT stderr start_thread+0x104 (0x00003FFF7BCC8CD4 [libpthread.so.0+0x8cd4])
DLT stderr clone+0xe4 (0x00003FFF7BBC7F14 [libc.so.6+0x127f14])
DLT stderr ---------------------------------------
DLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/03/01 01:54:00 - please wait.
DLT stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/core.20220301.015400.9054.0001.dmp' in response to an event
DLT stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/core.20220301.015400.9054.0001.dmp
DLT stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/javacore.20220301.015400.9054.0002.txt' in response to an event
STF 01:54:01.136 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/javacore.20220301.015400.9054.0002.txt
STF 01:54:01.136 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/core.20220301.015400.9054.0001.dmp
DLT stderr javacore file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/javacore.20220301.015400.9054.0002.txt
DLT stderr core file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/core.20220301.015400.9054.0001.dmp
DLT stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/javacore.20220301.015400.9054.0002.txt
DLT stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/Snap.20220301.015400.9054.0003.trc' in response to an event
STF 01:54:02.979 - Found dump at: /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/Snap.20220301.015400.9054.0003.trc
DLT stderr Snap file generated - /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/Snap.20220301.015400.9054.0003.trc
DLT stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/Snap.20220301.015400.9054.0003.trc
DLT stderr JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/jitdump.20220301.015400.9054.0004.dmp' in response to an event
DLT stderr JVMDUMP051I JIT dump occurred in '(unnamed thread)' thread 0x0000000000095700
DLT stderr JVMDUMP053I JIT dump is recompiling java/lang/J9VMInternals.fastIdentityHashCode(Ljava/lang/Object;)I
DLT stderr JVMDUMP053I JIT dump is recompiling java/lang/J9VMInternals.fastIdentityHashCode(Ljava/lang/Object;)I
DLT stderr JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal_testList_0/aqa-tests/TKG/output_16461281238347/DaaLoadTest_daa1_special_5m_21/20220301-015355-DaaLoadTest/results/jitdump.20220301.015400.9054.0004.dmp
DLT stderr JVMDUMP013I Processed dump event "gpf", detail "".
STF 01:54:05.741 - **FAILED** Process DLT ended with exit code (255) and not the expected exit code/s (0)

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal/840/, failure in testSCCMLTests1_openj9_1 and cmdLineTester_SCHelperCompatTests_unix_0. Both fail in a number of sub-tests, all due to success and failure conditions not being found.

Console log excerpt for testSCCMLTests1_openj9_1.

Testing: Test 43: LIR 1445.1 : Debug Area PrintStats Test : Ensure debug area is 3 MB in size and contains data
Test start time: 2022/03/01 02:43:24 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_ppc64le_linux_jit_Personal_testList_0/openjdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=ShareClassesCMLTests,printStats -version
Time spent starting: 19 milliseconds
Time spent executing: 59 milliseconds
Test result: FAILED
 [ERR] JVMSHRC023E Cache does not exist
 [ERR] JVMSHRC023E Cache does not exist
>> Success condition was not found: [Output match: Class debug area size[\s]*= 3145728]
>> Failure condition was not found: [Output match: Class debug area size[\s]*= 0]
>> Failure condition was not found: [Output match: Class debug area used bytes[\s]*= 0]
>> Failure condition was not found: [Output match: Class debug area % used[\s]*= 0%]
>> Failure condition was not found: [Output match: Unhandled Exception]
>> Failure condition was not found: [Output match: Exception:]
>> Failure condition was not found: [Output match: corrupt]
>> Failure condition was not found: [Output match: Processing dump event]

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/276, failure in:

The failures are similar to those reported in another comment in this thread; they're a mix of segfaults and JNI error/string index out of bounds.

One new type of failure:

MauveMultiThrdLoad_5m_0

STF 01:05:34.670 - Monitoring processes: LT 
LT  java.lang.RuntimeException: Unable to parse --add-opens <module>/<package>: java.base/java.lang
STF 01:05:34.908 - **FAILED** Process LT  ended with exit code (1) and not the expected exit code/s (0)
mpirvu commented 2 years ago

Those JLM test failures are happening for quite a while. I don't think they are JITServer related. They should receive lower priority anyway.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_extended.system_x86-64_linux_jit_Personal/200/, various failures. I've sorted them below. Haven't been able to reproduce.

Unexpected crash (no other detail in the log)

Segfault

Null pointer exception

All the null pointer exceptions look like this, from SharedClasses.SCM01.SingleCL_1:

SCL4 stderr java.lang.NullPointerException
SCL4 stderr     at java.base/com.ibm.jit.JITHelpers.getClassDepthAndFlagsFromJ9Class64(Native Method)
SCL4 stderr     at java.base/com.ibm.jit.JITHelpers.isArray(JITHelpers.java:141)
SCL4 stderr     at java.base/java.lang.J9VMInternals.fastIdentityHashCode(J9VMInternals.java:408)
SCL4 stderr     at java.base/java.lang.Object.hashCode(Object.java:120)
SCL4 stderr     at java.base/java.util.WeakHashMap.hash(WeakHashMap.java:303)
SCL4 stderr     at java.base/java.util.WeakHashMap.put(WeakHashMap.java:455)
SCL4 stderr     at java.base/java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:171)
SCL4 stderr     at net.openj9.test.sc.LoaderSlave.run(LoaderSlave.java:68)
SCL4 stderr     at net.openj9.test.sc.LoaderSlave.main(LoaderSlave.java:52)
cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/278/, failure in cmdLineTester_jvmtitests_hcr_OSRG_nongold_4, I think. The TAP results couldn't be parsed because that failure seems to have been corrupted enough to contain embedded NULLs. There were a couple of lines in the parse error that seemed intact:

        Exception in thread "Attach API initializer" java.lang.ExceptionInInitializerError

        Error: A JNI error has occurred, please check your installation and try again

but most of the location information has been destroyed.

cjjdespres commented 2 years ago

I think that's the backlog all sorted, so I'll start looking at some of these in more detail.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/267/, failure in jdk_lang_1. A segfault.

Console log:

06:27:33          TestFloatingDecimal
06:27:33  stderr:
06:27:33  Unhandled exception
06:27:33  Type=Segmentation error vmState=0x00000000
06:27:33  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
06:27:33  Handler1=00003FFFA2AE4770 Handler2=00003FFFA2A0C900
06:27:33  R0=00003FFFA2B81EB8 R1=00003FFF3F0EB2E0 R2=00003FFFA2CF6E00 R3=0000000000000004
06:27:33  R4=00003FFF75A77650 R5=00003FFF9C474900 R6=00003FFF2570C3E0 R7=000000000000049C
06:27:33  R8=0000000000000018 R9=0000000000000018 R10=0000000000000008 R11=00003FFF75A05E78
06:27:33  R12=0000000000004400 R13=00003FFF3F0F6900 R14=00003FFEF8008EC0 R15=00003FFF9C557600
06:27:33  R16=00003FFF3FB80038 R17=FFFFFFFFFFFFFFFF R18=00003FFF3F0EC400 R19=00003FFF2570C76A
06:27:33  R20=00003FFF9C3DA1B8 R21=0000000000090000 R22=00003FFEF8008E90 R23=00003FFF9C5559D0
06:27:33  R24=00003FFF9C557600 R25=0000000000000000 R26=00003FFF3F0EB640 R27=00003FFF3F0EB520
06:27:33  R28=00003FFF3F0EB500 R29=00003FFF75A6BF58 R30=0000000000000000 R31=0000000000000000
06:27:33  NIP=00003FFFA2B7AD38 MSR=800000010280F033 ORIG_GPR3=C0000000000027E0 CTR=00003FFFA2B835B8
06:27:33  LINK=00003FFFA2B81EB8 XER=0000000000000000 CCR=0000000042004222 SOFTE=0000000000000001
06:27:33  TRAP=0000000000000300 DAR=0000000000000020 dsisr=0000000040000000 RESULT=0000000000000000
06:27:33  FPR0 00003ffef8008f08 (f: 4160786176.000000, d: 3.476459e-310)
06:27:33  FPR1 405472d940000000 (f: 1073741824.000000, d: 8.179451e+01)
06:27:33  FPR2 00003fff3f0e6d30 (f: 1057910080.000000, d: 3.476518e-310)
06:27:33  FPR3 3fc7466496cb03de (f: 2529887232.000000, d: 1.818357e-01)
06:27:33  FPR4 3fcc71c51d8e78af (f: 495876256.000000, d: 2.222220e-01)
06:27:33  FPR5 3fd2492494229359 (f: 2485293824.000000, d: 2.857143e-01)
06:27:33  FPR6 3fe5555560000000 (f: 1610612736.000000, d: 6.666667e-01)
06:27:33  FPR7 3f2a1e97e0000000 (f: 3758096384.000000, d: 1.992760e-04)
06:27:33  FPR8 4036000000000000 (f: 0.000000, d: 2.200000e+01)
06:27:33  FPR9 3fd99c6800000000 (f: 0.000000, d: 4.001713e-01)
06:27:33  FPR10 3fa9dd9200000000 (f: 0.000000, d: 5.051857e-02)
06:27:33  FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
06:27:33  FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
06:27:33  FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
06:27:33  Module_base_address=00003FFFA2AA0000
06:27:33  Target=2_90_20220302_953 (Linux 3.10.0-1160.53.1.el7.ppc64le)
06:27:33  CPU=ppc64le (4 logical CPUs) (0xde750000 RAM)
06:27:33  ----------- Stack Backtrace -----------
06:27:33  protectedBacktrace+0x24 (0x00003FFFA2A06B04 [libj9prt29.so+0x36b04])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  omrintrospect_backtrace_thread_raw+0xe4 (0x00003FFFA2A07134 [libj9prt29.so+0x37134])
06:27:33  protectedIntrospectBacktraceThread+0x24 (0x00003FFFA2A067E4 [libj9prt29.so+0x367e4])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  omrintrospect_backtrace_thread+0x90 (0x00003FFFA2A068E0 [libj9prt29.so+0x368e0])
06:27:33  generateDiagnosticFiles+0x140 (0x00003FFFA2AE45F0 [libj9vm29.so+0x445f0])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  structuredSignalHandler+0x1b4 (0x00003FFFA2AE4924 [libj9vm29.so+0x44924])
06:27:33  mainSynchSignalHandler+0x260 (0x00003FFFA2A0CB60 [libj9prt29.so+0x3cb60])
06:27:33  __kernel_sigtramp_rt64+0x0 (0x00003FFFA3410478)
06:27:33  bytecodeLoopFull+0x8948 (0x00003FFFA2B81EB8 [libj9vm29.so+0xe1eb8])
06:27:33   (0x00003FFFA2C162A0 [libj9vm29.so+0x1762a0])
06:27:33  runJavaThread+0x340 (0x00003FFFA2AB93D0 [libj9vm29.so+0x193d0])
06:27:33  javaProtectedThreadProc+0xf0 (0x00003FFFA2B55750 [libj9vm29.so+0xb5750])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  javaThreadProc+0x64 (0x00003FFFA2B50984 [libj9vm29.so+0xb0984])
06:27:33  thread_wrapper+0x1a8 (0x00003FFFA29A13E8 [libj9thr29.so+0x113e8])
06:27:33  start_thread+0x104 (0x00003FFFA3388CD4 [libpthread.so.0+0x8cd4])
06:27:33  clone+0xe4 (0x00003FFFA3247F14 [libc.so.6+0x127f14])
06:27:33  ---------------------------------------
06:27:33  JVMDUMP039I Processing dump event "gpf", detail "" at 2022/03/02 03:06:39 - please wait.
06:27:33  
06:27:33  
06:27:33  *** Invalid JIT return address 00003FFF257AEFEB in 00003FFF3E9F98C0
06:27:33  
06:27:33  Unhandled exception
06:27:33  Type=Segmentation error vmState=0x0002000f
06:27:33  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
06:27:33  Handler1=00003FFFA2AE4770 Handler2=00003FFFA2A0C900
06:27:33  R0=00003FFFA2B3DB90 R1=00003FFF3E9F9790 R2=00003FFFA2CF6E00 R3=0000000000000001
06:27:33  R4=00003FFFFFFFFFFE R5=00003FFEF8008EF0 R6=0000000000000000 R7=0000000000000000
06:27:33  R8=00003FFF9C67D188 R9=00003FFFFFFFFFFE R10=00003FFF759FEB80 R11=000000000000000F
06:27:33  R12=00003FFFA1908620 R13=00003FFF3EA06900 R14=00003FFEF002D680 R15=00003FFEEC169600
06:27:33  R16=00003FFF3FB80038 R17=00003FFF95718428 R18=00003FFEF002DA08 R19=0000000000000000
06:27:33  R20=0000000000000296 R21=0000000000000297 R22=0000000000000000 R23=0000000000020019
06:27:33  R24=00003FFFA1761910 R25=0000000000000000 R26=00003FFEF8008F10 R27=0000000004600002
06:27:33  R28=00003FFFA2CF31E0 R29=00003FFEEC169600 R30=0000000000000000 R31=00003FFF3E9F98C0
06:27:33  NIP=00003FFFA2B3DE78 MSR=800000010280F033 ORIG_GPR3=C0000000000027E0 CTR=00003FFFA1908620
06:27:33  LINK=00003FFFA2B3DB90 XER=0000000020000000 CCR=0000000024004844 SOFTE=0000000000000001
06:27:33  TRAP=0000000000000300 DAR=0000400000000006 dsisr=0000000040000000 RESULT=0000000000000000
06:27:33  FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR1 3fefae147ae147ae (f: 2061584256.000000, d: 9.900000e-01)
06:27:33  FPR2 3fd34413509f79fb (f: 1352628736.000000, d: 3.010300e-01)
06:27:33  FPR3 c000000000000000 (f: 0.000000, d: -2.000000e+00)
06:27:33  FPR4 3fcc71c51d8e78af (f: 495876256.000000, d: 2.222220e-01)
06:27:33  FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR6 3fd999999997fa04 (f: 2576873984.000000, d: 4.000000e-01)
06:27:33  FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR8 37fc801880000000 (f: 2147483648.000000, d: 5.234692e-39)
06:27:33  FPR9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR10 0037003600390035 (f: 3735605.000000, d: 1.279463e-307)
06:27:33  FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
06:27:33  FPR12 405c400000000000 (f: 0.000000, d: 1.130000e+02)
06:27:33  FPR13 4073400000000000 (f: 0.000000, d: 3.080000e+02)
06:27:33  FPR14 3fe7e9be40548f8e (f: 1079283584.000000, d: 7.472831e-01)
06:27:33  FPR15 3fc9f35e32b8b498 (f: 850965632.000000, d: 2.027395e-01)
06:27:33  FPR16 3feae41b4b4cc3de (f: 1263322112.000000, d: 8.403450e-01)
06:27:33  FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
06:27:33  Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
06:27:33  Module_base_address=00003FFFA2AA0000
06:27:33  Target=2_90_20220302_953 (Linux 3.10.0-1160.53.1.el7.ppc64le)
06:27:33  CPU=ppc64le (4 logical CPUs) (0xde750000 RAM)
06:27:33  ----------- Stack Backtrace -----------
06:27:33  protectedBacktrace+0x24 (0x00003FFFA2A06B04 [libj9prt29.so+0x36b04])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  omrintrospect_backtrace_thread_raw+0xe4 (0x00003FFFA2A07134 [libj9prt29.so+0x37134])
06:27:33  protectedIntrospectBacktraceThread+0x24 (0x00003FFFA2A067E4 [libj9prt29.so+0x367e4])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  omrintrospect_backtrace_thread+0x90 (0x00003FFFA2A068E0 [libj9prt29.so+0x368e0])
06:27:33  generateDiagnosticFiles+0x140 (0x00003FFFA2AE45F0 [libj9vm29.so+0x445f0])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  structuredSignalHandler+0x1b4 (0x00003FFFA2AE4924 [libj9vm29.so+0x44924])
06:27:33  mainSynchSignalHandler+0x260 (0x00003FFFA2A0CB60 [libj9prt29.so+0x3cb60])
06:27:33  __kernel_sigtramp_rt64+0x0 (0x00003FFFA3410478)
06:27:33  walkStackFrames+0x7f0 (0x00003FFFA2B3DB90 [libj9vm29.so+0x9db90])
06:27:33  _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x88 (0x00003FFFA14A57B8 [libj9gc_full29.so+0x557b8])
06:27:33  _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x148 (0x00003FFFA149A958 [libj9gc_full29.so+0x4a958])
06:27:33  _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0x10c (0x00003FFFA1498E4C [libj9gc_full29.so+0x48e4c])
06:27:33  _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0xa0 (0x00003FFFA149CD70 [libj9gc_full29.so+0x4cd70])
06:27:33  _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x318 (0x00003FFFA1639048 [libj9gc_full29.so+0x1e9048])
06:27:33  _ZN23MM_ParallelScavengeTask3runEP18MM_EnvironmentBase+0x1c (0x00003FFFA16AFD9C [libj9gc_full29.so+0x25fd9c])
06:27:33  _ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0x130 (0x00003FFFA15D1DB0 [libj9gc_full29.so+0x181db0])
06:27:33  _ZN12MM_Scavenger8scavengeEP18MM_EnvironmentBase+0xcc (0x00003FFFA161EE0C [libj9gc_full29.so+0x1cee0c])
06:27:33  _ZN12MM_Scavenger24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x2d4 (0x00003FFFA16304C4 [libj9gc_full29.so+0x1e04c4])
06:27:33  _ZN12MM_Scavenger22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0xbcc (0x00003FFFA163185C [libj9gc_full29.so+0x1e185c])
06:27:33  _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0x13c (0x00003FFFA15A155C [libj9gc_full29.so+0x15155c])
06:27:33  _ZN26MM_MemorySubSpaceSemiSpace23allocationRequestFailedEP18MM_EnvironmentBaseP22MM_AllocateDescriptionN17MM_MemorySubSpace14AllocationTypeEP28MM_ObjectAllocationInterfacePS4_S8_+0x42c (0x00003FFFA16AACAC [libj9gc_full29.so+0x25acac])
06:27:33  _ZN24MM_MemorySubSpaceGeneric11allocateTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP28MM_ObjectAllocationInterfaceP17MM_MemorySubSpaceS7_b+0x444 (0x00003FFFA1687B24 [libj9gc_full29.so+0x237b24])
06:27:33  _ZN23MM_TLHAllocationSupport7refreshEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x520 (0x00003FFFA15DB910 [libj9gc_full29.so+0x18b910])
06:27:33  _ZN23MM_TLHAllocationSupport15allocateFromTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0xe4 (0x00003FFFA15DBB54 [libj9gc_full29.so+0x18bb54])
06:27:33  _ZN25MM_TLHAllocationInterface15allocateFromTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x28 (0x00003FFFA15DA5B8 [libj9gc_full29.so+0x18a5b8])
06:27:33  _ZN25MM_TLHAllocationInterface14allocateObjectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x1e8 (0x00003FFFA15DA7E8 [libj9gc_full29.so+0x18a7e8])
06:27:33  _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x100 (0x00003FFFA15E28C0 [libj9gc_full29.so+0x1928c0])
06:27:33  J9AllocateObject+0x1a0 (0x00003FFFA14ACD80 [libj9gc_full29.so+0x5cd80])
06:27:33  old_slow_jitNewObject+0x398 (0x00003FFFA232F1B8 [libj9jit29.so+0xbaf1b8])
06:27:33   (0x00003FFFA234D2A8 [libj9jit29.so+0xbcd2a8])
06:27:33  sidecarInvokeReflectMethodImpl+0x3c0 (0x00003FFFA2ABE700 [libj9vm29.so+0x1e700])
06:27:33  sidecarInvokeReflectMethod+0x38 (0x00003FFFA2ABFB98 [libj9vm29.so+0x1fb98])
06:27:33  JVM_InvokeMethod_Impl+0xb4 (0x00003FFFA11C11E4 [libjclse29.so+0x811e4])
06:27:33  JVM_InvokeMethod+0x40 (0x00003FFFA3014450 [libjvm.so+0x24450])
06:27:33  JVM_InvokeMethod+0x34 (0x00003FFFA30E6CF4 [libjvm.so+0x6cf4])
06:27:33  Java_jdk_internal_reflect_NativeMethodAccessorImpl_invoke0+0x24 (0x00003FFF3F7BCF74 [libjava.so+0xcf74])
06:27:33   (0x00003FFF5CE675A0 [<unknown>+0x0])
06:27:33  runJavaThread+0x340 (0x00003FFFA2AB93D0 [libj9vm29.so+0x193d0])
06:27:33  javaProtectedThreadProc+0xf0 (0x00003FFFA2B55750 [libj9vm29.so+0xb5750])
06:27:33  omrsig_protect+0x358 (0x00003FFFA2A0DD08 [libj9prt29.so+0x3dd08])
06:27:33  javaThreadProc+0x64 (0x00003FFFA2B50984 [libj9vm29.so+0xb0984])
06:27:33  thread_wrapper+0x1a8 (0x00003FFFA29A13E8 [libj9thr29.so+0x113e8])
06:27:33  start_thread+0x104 (0x00003FFFA3388CD4 [libpthread.so.0+0x8cd4])
06:27:33  clone+0xe4 (0x00003FFFA3247F14 [libc.so.6+0x127f14])
06:27:33  ---------------------------------------
06:27:33  JVMDUMP039I Processing dump event "gpf", detail "" at 2022/03/02 03:06:39 - please wait.
06:27:33  
06:27:33  TEST RESULT: Error. Agent error: java.lang.Exception: Agent 11 timed out with a timeout of 960 seconds; check console log for any additional details
06:27:33  --------------------------------------------------
06:29:30  Test results: passed: 726; error: 1
06:29:53  Report written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/jvmtest/openjdk/report/html/report.html
06:29:53  Results written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_jit_Personal/aqa-tests/TKG/output_16462121344783/jdk_lang_1/work
06:29:53  Error: Some tests failed or other problems occurred.
06:29:53  
06:29:53  jdk_lang_1_FAILED

I think this may already be tracked in #14182, though in that issue the failure is in jdk_lang_0 and not jdk_lang_1.

mpirvu commented 2 years ago

I think this may already be tracked in https://github.com/eclipse-openj9/openj9/issues/14182, though in that issue the failure is in jdk_lang_0 and not jdk_lang_1.

yes, I think this is the same issue, thus not JITServer specific

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_special.system_ppc64le_linux_jit_Personal/193/, failures in

A mix of segfaults and null java.home errors as in this job, except for DaaLoadTest_daa1_special_5m_21. The output of that test is:

DLT stderr Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.logging.log4j.message.DefaultFlowMessageFactory
DLT stderr  at org.apache.logging.log4j.spi.AbstractLogger.<clinit>(AbstractLogger.java:101)
DLT stderr  at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:61)
DLT stderr  at net.adoptopenjdk.loadTest.LoadTest.<clinit>(LoadTest.java:46)
DLT stderr Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.DefaultFlowMessageFactory
DLT stderr  at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:827)
DLT stderr  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
DLT stderr  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1091)

which was observed in this job.

The failure in ParallelStreamsLoadTest_special_J9_13 is the second "null property: java.home" failure in this test.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.system_ppc64le_linux_jit_Personal/277/, various failures.

cjjdespres commented 2 years ago

I'll be updating the summary comment, but I haven't been able to reproduce any of the failures in

which were most of the larger test failures from a few days ago. I also haven't been able to reproduce the DaaLoadTest_all_special_5m_21 without JITAAS yet, so I'll look at that one a bit more.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/898/, failures in:

All of them are segfaults with the same stack trace. Example from cmdLineTester_jvmtitests_debug_10:

===============================================
Running test cmdLineTester_jvmtitests_debug_10 ...
===============================================
cmdLineTester_jvmtitests_debug_10 Start Time: Thu Mar  3 02:07:44 2022 Epoch Time (ms): 1646302064347
variation: Mode107-OSR
JVM_OPTIONS: -XX:+UseJITServer -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,count=0
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007FDC34F89970 Handler2=00007FDC34CE70E0 InaccessibleAddress=0000000000000000
RDI=00000000C55918E2 RSI=00000000C263B5B0 RAX=00000000C263B738 RBX=000000000000002D
RCX=0000000000046A00 RDX=00000000C5591949 R8=00000000C559190F R9=00000000C262FBF8
R10=0000000000000046 R11=0000000000000007 R12=00000000000003C0 R13=00007FDC30000AA0
R14=00000000000003E0 R15=FFFFFFFFFFFFFFB0
RIP=0000000000000000 GS=0000 FS=0000 RSP=0000000000109428
EFlags=0000000000010202 CS=0033 RBP=0000000000016F00 ERR=0000000000000014
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 00000000000f2ca0 (f: 994464.000000, d: 4.913305e-318)
xmm1 000000003e3ab283 (f: 1044034176.000000, d: 5.158214e-315)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 bfe025fb8ffd0aee (f: 2415725312.000000, d: -5.046366e-01)
xmm4 bfb1973c5a611ccc (f: 1516313856.000000, d: -6.871392e-02)
xmm5 0000000000039d08 (f: 236808.000000, d: 1.169987e-318)
xmm6 00007fdc3534bf80 (f: 892649344.000000, d: 6.945761e-310)
xmm7 0000000000000005 (f: 5.000000, d: 2.470328e-323)
xmm8 616c2f6176616a4c (f: 1986095744.000000, d: 1.981285e+161)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Target=2_90_20220303_1036 (Linux 4.15.0-154-generic)
CPU=amd64 (4 logical CPUs) (0xf6813000 RAM)
----------- Stack Backtrace -----------
protectedBacktrace+0x12 (0x00007FDC34CE3682 [libj9prt29.so+0x26682])
omrsig_protect+0x1e3 (0x00007FDC34CE7E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread_raw+0xbf (0x00007FDC34CE3B7F [libj9prt29.so+0x26b7f])
omrsig_protect+0x1e3 (0x00007FDC34CE7E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread+0x70 (0x00007FDC34CE3540 [libj9prt29.so+0x26540])
generateDiagnosticFiles+0x84 (0x00007FDC34F89474 [libj9vm29.so+0x3f474])
omrsig_protect+0x1e3 (0x00007FDC34CE7E43 [libj9prt29.so+0x2ae43])
vmSignalHandler+0x185 (0x00007FDC34F896E5 [libj9vm29.so+0x3f6e5])
 (0x00007FDC350868A5 [libj9vm29.so+0x13c8a5])

I've run job/Grinder/21524/ with JITAAS and quite a few of the same failures, and job/Grinder/21525/ without JITAAS and no failures. I don't know if this is significant, but the failures were very unevenly distributed in the iterations, with three failing heavily and two passing.

cjjdespres commented 2 years ago

From job/Test_openjdk17_j9_extended.functional_ppc64le_linux_jit_Personal/199/, failure in testSCCacheManagement_0. "Invalid number of cache found".

Console log:

1) testSharedCacheJavaAPI(tests.sharedclasses.options.junit.TestOptionsDefault)tests.sharedclasses.TestFailedException: SharedClassUtilities.getSharedCacheInfo failed: Invalid number of cache found   expected: 7 found: 8
 oldCacheCount is 0
 persistentGroupAccessCount is 1
 nonpersistentGroupAccessCount is 1
 persistentCount is 3
 nonpersistentCount is 1
 snapshotCount is 1
    at tests.sharedclasses.TestUtils.fail(TestUtils.java:598)
    at tests.sharedclasses.options.TestSharedCacheJavaAPI.main(TestSharedCacheJavaAPI.java:208)
    at tests.sharedclasses.options.junit.TestOptionsBase.testSharedCacheJavaAPI(TestOptionsBase.java:127)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Haven't attempted to reproduce.

cjjdespres commented 2 years ago

From job/Test_openjdk8_j9_extended.functional_x86-64_linux_jit_Personal/281/, failures in

Both segmentation faults with similar stack traces. Console log for testDefaultDisclaimMemory_2

Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F38D7432920 Handler2=00007F38D71910E0 InaccessibleAddress=0000000000000000
RDI=00000000C007FB68 RSI=00000000C036F4A0 RAX=00000000C007FB68 RBX=00000000001CA000
RCX=00000000C037F5A8 RDX=0000000000000016 R8=0000000000000001 R9=0000000000000073
R10=0000000000000003 R11=0000000000000001 R12=00000000C03456F0 R13=00000000C0345730
R14=0000000011C232E6 R15=00000000C02D33C8
RIP=0000000000000000 GS=0000 FS=0000 RSP=00000000000E4578
EFlags=0000000000010202 CS=0033 RBP=0000000000016F00 ERR=0000000000000014
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 00000000001c90f8 (f: 1872120.000000, d: 9.249502e-318)
xmm1 00000000c037f5a8 (f: 3224892928.000000, d: 1.593309e-314)
xmm2 00007f38d7fcc830 (f: 3623667712.000000, d: 6.911095e-310)
xmm3 bfe014068d2a48c8 (f: 2368358656.000000, d: -5.024445e-01)
xmm4 000000003f6d41de (f: 1064124864.000000, d: 5.257476e-315)
xmm5 bff0000000000000 (f: 0.000000, d: -1.000000e+00)
xmm6 bfb1973c5a611ccc (f: 1516313856.000000, d: -6.871392e-02)
xmm7 0000000000000005 (f: 5.000000, d: 2.470328e-323)
xmm8 01000a0000002100 (f: 8448.000000, d: 7.308923e-304)
xmm9 ff0000ff0000ffff (f: 65535.000000, d: -5.487458e+303)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Target=2_90_20220303_978 (Linux 5.4.0-91-generic)
CPU=amd64 (4 logical CPUs) (0xf5fa5000 RAM)
----------- Stack Backtrace -----------
protectedBacktrace+0x12 (0x00007F38D718D682 [libj9prt29.so+0x26682])
omrsig_protect+0x1e3 (0x00007F38D7191E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread_raw+0xbf (0x00007F38D718DB7F [libj9prt29.so+0x26b7f])
omrsig_protect+0x1e3 (0x00007F38D7191E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread+0x70 (0x00007F38D718D540 [libj9prt29.so+0x26540])
generateDiagnosticFiles+0x84 (0x00007F38D7432464 [libj9vm29.so+0x3e464])
omrsig_protect+0x1e3 (0x00007F38D7191E43 [libj9prt29.so+0x2ae43])
vmSignalHandler+0x185 (0x00007F38D74326D5 [libj9vm29.so+0x3e6d5])
 (0x00007F38D7526155 [libj9vm29.so+0x132155])

Console log for TestFileLocking_SE80_0

Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F77B42F3920 Handler2=00007F77B40520E0 InaccessibleAddress=0000000000000000
RDI=00000000001B3700 RSI=00000000FFEA9B98 RAX=0000000085D75480 RBX=0000000000000001
RCX=0000000085E4D208 RDX=0000000000000016 R8=0000000000000001 R9=0000000000000073
R10=0000000085D75480 R11=00007F7794555F68 R12=00000000FFE82B70 R13=0000000000049F00
R14=0000000000049F00 R15=00000000FFEA3D88
RIP=0000000000000000 GS=0000 FS=0000 RSP=00000000000E4578
EFlags=0000000000010206 CS=0033 RBP=0000000000016F00 ERR=0000000000000014
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 00000000001b2788 (f: 1779592.000000, d: 8.792353e-318)
xmm1 4c293b7463656a62 (f: 1667590784.000000, d: 7.919268e+58)
xmm2 00007f77b6343850 (f: 3056875520.000000, d: 6.924435e-310)
xmm3 0000000033e8a4cd (f: 870884544.000000, d: 4.302741e-315)
xmm4 000000003c949efd (f: 1016372992.000000, d: 5.021550e-315)
xmm5 00071d003950f4bf (f: 961606848.000000, d: 9.892240e-309)
xmm6 000000003d1677de (f: 1024882624.000000, d: 5.063593e-315)
xmm7 2f65636170736b72 (f: 1886612352.000000, d: 2.254787e-80)
xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm9 000000003e17cee7 (f: 1041747712.000000, d: 5.146917e-315)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 0000ff0000000000 (f: 0.000000, d: 1.385239e-309)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Target=2_90_20220303_978 (Linux 3.10.0-1160.59.1.el7.x86_64)
CPU=amd64 (4 logical CPUs) (0x1e8d07000 RAM)
----------- Stack Backtrace -----------
protectedBacktrace+0x12 (0x00007F77B404E682 [libj9prt29.so+0x26682])
omrsig_protect+0x1e3 (0x00007F77B4052E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread_raw+0xbf (0x00007F77B404EB7F [libj9prt29.so+0x26b7f])
omrsig_protect+0x1e3 (0x00007F77B4052E43 [libj9prt29.so+0x2ae43])
omrintrospect_backtrace_thread+0x70 (0x00007F77B404E540 [libj9prt29.so+0x26540])
generateDiagnosticFiles+0x84 (0x00007F77B42F3464 [libj9vm29.so+0x3e464])
omrsig_protect+0x1e3 (0x00007F77B4052E43 [libj9prt29.so+0x2ae43])
vmSignalHandler+0x185 (0x00007F77B42F36D5 [libj9vm29.so+0x3e6d5])
 (0x00007F77B43E7155 [libj9vm29.so+0x132155])

The grinder job/Grinder/21553/ with JITAAS reproduces TestFileLocking_SE80_0 with a 4/50 failure rate, but does not reproduce testDefaultDisclaimMemory_2. I have been unable to reproduce TestFileLocking_SE80_0 without JITAAS.

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_sanity.functional_x86-64_linux_jit_Personal/899/, various failures, all segfaults. The following have been observed and reproduced in https://github.com/eclipse-openj9/openj9/issues/14538#issuecomment-1059247150

and these ones are new:

cjjdespres commented 2 years ago

From job/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal/290/, failures in

Both fail because the tests fail to connect to the remote server that was created. Console log for testOSMXBeanRemote_1:

[TestOperatingSystemMXBean] [INFO] Remote Server stdout: [RemoteServer] [INFO] =========RemoteServer Starts!=========
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472562324 ms.  Retry 1 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472563366 ms.  Retry 2 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472564407 ms.  Retry 3 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472565491 ms.  Retry 4 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472566531 ms.  Retry 5 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472567572 ms.  Retry 6 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472568613 ms.  Retry 7 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472569654 ms.  Retry 8 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472570695 ms.  Retry 9 after 1000 ms.
[TestOperatingSystemMXBean] [WARN] Connection request failed: Connection refused to host: 9.24.248.182; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
[TestOperatingSystemMXBean] [WARN] Connection failed at: 1646472571736 ms.  Retry 10 after 1000 ms.
FAILED: runTestOSMXBean
java.lang.AssertionError: !!!Test(s) couldn't be started in remote mode: Connection setup failed !!! expected [true] but found [false]
    at org.testng.Assert.fail(Assert.java:96)
    at org.testng.Assert.failNotEquals(Assert.java:776)
    at org.testng.Assert.assertTrue(Assert.java:44)
    at org.openj9.test.management.TestOperatingSystemMXBean.runTestOSMXBean(TestOperatingSystemMXBean.java:195)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.TestNG.privateMain(TestNG.java:1354)
    at org.testng.TestNG.main(TestNG.java:1323)