Open pshipton opened 4 years ago
Started a 1x grinder https://ci.eclipse.org/openj9/job/Grinder/718/ - passed on ub16-390-3
2x grinders https://ci.eclipse.org/openj9/job/Grinder/719 ub16-390-3 - failed only java/nio/charset/coders/BashStreams.java.BashStreams, which looks like a dup of https://github.com/eclipse/openj9/issues/8652
https://ci.eclipse.org/openj9/job/Grinder/720 ub16-390-1 - passed https://ci.eclipse.org/openj9/job/Grinder/721 ub16-390-4 - passed https://ci.eclipse.org/openj9/job/Grinder/722 rh7-390-3 - passed
1x on rh7-390-4 https://ci.eclipse.org/openj9/job/Grinder/723/ - passed
Thought this could be related to #8992 but that looks to be a compile time crash and this is a runtime exception, so likely not related. @VermaSh could you take a look at this one given it is in the area related to Strings and we just merged the VSTRS fixes a few days ago. It could be related to that change.
@pshipton @jdekonin Could I get access to ub16-390-1
Could I get access to ub16-390-1
Provided instructions via PM on Slack
Grinder shows a 1/30 failure rate: https://ci.eclipse.org/openj9/job/Grinder/731
@VermaSh any updates?
@fjeremic Failures in that grinder seem to be agent related Agent communication error: java.io.EOFException; check console log for any additional details
There don't seem to be any functional failures. Running them locally on ub16-390-1
machine to replicate failures
@VermaSh I don't think so. Look at the console output and search for "PASSED: 0 FAILED: 1", there is clearly a crash here which looks similar to the original one:
13:00:08 Unexpected exception: x-JISAutoDetect NullPointerException "??"[0/5] => null ""[0/19]
13:00:08 failures=5000
13:00:08 -----
13:00:08 Unexpected exception: x-JISAutoDetect NullPointerException "??"[0/5] => null ""[0/19] (direct)
13:00:08 failures=5000
13:00:08 -----
13:00:08 Unexpected exception: x-JISAutoDetect NullPointerException "??"[0/5] => null ""[0/4] (direct)
13:00:08 failures=1
13:00:08
13:00:08 Passed = 512806702, failed = 10001
13:00:08
13:00:08 STDERR:
13:00:08 java.lang.NullPointerException
13:00:08 at java.base/sun.util.PreHashedMap.get(PreHashedMap.java:139)
13:00:08 at java.base/sun.nio.cs.StandardCharsets.canonicalize(StandardCharsets.java:1227)
13:00:08 at java.base/sun.nio.cs.StandardCharsets.lookup(StandardCharsets.java:1304)
13:00:08 at java.base/sun.nio.cs.StandardCharsets.charsetForName(StandardCharsets.java:1338)
13:00:08 at java.base/java.nio.charset.Charset.lookup2(Charset.java:473)
13:00:08 at java.base/java.nio.charset.Charset.lookup(Charset.java:462)
13:00:08 at java.base/java.nio.charset.Charset.forName(Charset.java:526)
13:00:08 at jdk.charsets/sun.nio.cs.ext.JISAutoDetect$Decoder.decodeLoop(JISAutoDetect.java:146)
13:00:08 at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:576)
13:00:08 at FindDecoderBugs$CharsetTester.recode(FindDecoderBugs.java:206)
13:00:08 at FindDecoderBugs$CharsetTester.recode2(FindDecoderBugs.java:257)
13:00:08 at FindDecoderBugs$CharsetTester.test(FindDecoderBugs.java:279)
...
There are several more there with a final failure of a crash in GC due to what looks like a corrupt object.
ah dang, I thought Test Results section would show errors. Can you please launch a grinder using
OMR: git@github.com:VermaSh/omr.git
branch: jdk_nio_0
openj9: git@github.com:VermaSh/openj9.git
branch: jdk_nio_0
commit: ac8e0741103cca6fe77c61ed5e560cce2a91b1ed
There used to be a way to build personal builds off of branches, but I no longer see it here: https://ci.eclipse.org/openj9/view/all/
That list is absolutely massive. I tried kicking off a "Build_Personal" however it is unintuitive what you actually have to specify in those textboxes: https://ci.eclipse.org/openj9/view/Build_Personal/job/Build_JDK11_s390x_linux_Personal/
@pshipton do you might know of an easy way to get a build against the above repos/branches? We need to get a build so we can launch a grinder against it.
@fjeremic it doesn't have the best name, but the "personal build" job you are looking for is https://ci.eclipse.org/openj9/view/Pipelines/job/Pipeline-Release-Build/
Build with the requested branches: https://ci.eclipse.org/openj9/job/Build_JDK11_s390x_linux_Personal/537/
Grinder against it: https://ci.eclipse.org/openj9/job/Grinder/755/
Previous grinder failed 4/60, so failure rate is roughly 1/15. It looks like we fail only with the NPE, and not the other failures. I suspect we may be looking at different issues here. It is possible the NPE issue is not related to the other two exceptions. Regardless we need to keep tracking.
@fjeremic Thanks for launching the grinder. Can you launch one last grinder without my initial VSTRS changes
OMR: https://github.com/VermaSh/omr.git
branch: jdk_nio_0
openj9: https://github.com/VermaSh/openj9.git
branch: jdk_nio_0
commit: cb034a8d0d8d630638e94dcb558c5ee364c3f2c0
Unfortunately I can't specify individual commits, the pipeline only lets me specify branches. Can you create a new branch which has cb034a8d0d8d630638e94dcb558c5ee364c3f2c0
as the HEAD?
no worries. cb034a8d0d8d630638e94dcb558c5ee364c3f2c0
is the head in jdk_nio_0
so launching a build on that branch should be fine
@VermaSh the branches now result in a build failure. Likely a mismatch between OMR and OpenJ9: https://ci.eclipse.org/openj9/job/Build_JDK11_s390x_linux_Personal/539/console
Fixed the issue, uintptrj_t
got replaced with uintptr_t
@fjeremic can you please launch another build on the same branch
Build with the requested branches: https://ci.eclipse.org/openj9/job/Build_JDK11_s390x_linux_Personal/540
Grinder against it: https://ci.eclipse.org/openj9/job/Grinder/759/
A minor update:
Had some issues trying to get the test running locally on ub16-390-1
. Those have been resolved. Now I am running a grinder locally to reproduce the failures
Update:
I was able to successfully replicate failures through a local grinder for jdk_nio_0
, see list 1. Out of the tests in list 1 only java/nio/MappedByteBuffer/Truncate.java
failed when ran as a jdk_custom
target. I haven't had a chance to investigate further into the failures as I have been working on issues relating J9ZERZ10 and converter benchmark.
List 1: Test failures when running jdk_nio_0 grinder locally on ub16-390-1
java/nio/MappedByteBuffer/Truncate.java
java/nio/charset/Charset/DefaultCharsetTest.java
java/nio/charset/coders/SJISMappingPropTest.java
java/nio/charset/spi/CharsetProviderBasicTest.java
java/nio/file/Files/BytesAndLines.java
java/nio/file/Files/ReadWriteString.java
jdk/nio/zipfs/DirectoryStreamTests.java
jdk/nio/zipfs/InvalidZipHeaderTests.java
jdk/nio/zipfs/MultiReleaseJarTest.java
jdk/nio/zipfs/UpdateEntryTest.java
jdk/nio/zipfs/jarfs/JFSTester.java
sun/nio/cs/FindDecoderBugs.java
sun/nio/cs/FindOneCharEncoderBugs.java
sun/nio/cs/ISCIITest.java
sun/nio/cs/ISO8859x.java
sun/nio/cs/JISAutoDetectTest.java
sun/nio/cs/LatinCharReplacementTWTest.java
sun/nio/cs/LeftOverSurrogate.java
sun/nio/cs/NIOJISAutoDetectTest.java
sun/nio/cs/SJISCanEncode.java
sun/nio/cs/StreamEncoderClose.java
sun/nio/cs/SurrogateGB18030Test.java
sun/nio/cs/SurrogateTestEUCTW.java
sun/nio/cs/SurrogateTestHKSCS.java
sun/nio/cs/Test4206507.java
sun/nio/cs/TestConverterDroppedCharacters.java
sun/nio/cs/TestCp834_SBCS.java
sun/nio/cs/TestCp93xSISO.java
sun/nio/cs/TestEBCDICLineFeed.java
sun/nio/cs/TestEUC_TW.java
sun/nio/cs/TestGB18030.java
sun/nio/cs/TestHKSCS.java
sun/nio/cs/TestIBMBugs.java
sun/nio/cs/TestISO2022CNDecoder.java
sun/nio/cs/TestISO2022JP.java
sun/nio/cs/TestISO2022JPEncoder.java
sun/nio/cs/TestISO2022JPSubBytes.java
sun/nio/cs/TestIllegalISO2022Esc.java
sun/nio/cs/TestIllegalSJIS.java
sun/nio/cs/TestJIS0208Decoder.java
sun/nio/cs/TestJIS0212Decoder.java
sun/nio/cs/TestMS5022X.java
sun/nio/cs/TestMiscEUC_JP.java
sun/nio/cs/TestSJIS0213.java
sun/nio/cs/TestSJIS0213_SM.java
I am unable to get a log or a core dump for java/nio/MappedByteBuffer/Truncate.java
failure. I think that could have been due to this JVMCDRT000E Unable to locate JIT stack map - aborting VM
Going to try getting a log and core file for sun/nio/cs/FindOneCharEncoderBugs.java
Test output:
rerun:
cd /tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0 && \
HOME=/home/jenkins \
LANG=en_US.UTF-8 \
PATH=/bin:/usr/bin:/usr/sbin \
/tmp/sverma/docker_jdk11/openj9-openjdk-jdk11/build/linux-s390x-normal-server-release/images/jdk/bin/javac \
-J-ea \
-J-esa \
-J-Xmx512m \
-J-Xshareclasses:cacheDir=/tmp/sverma/jdk_custom \
-J-Xjit:count=0 \
-J-Djava.library.path=/tmp/sverma/docker_jdk11/openj9-openjdk-jdk11/build/linux-s390x-normal-server-release/images/test/jdk/jtreg/native \
-J-Dtest.class.path.prefix=/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/classes/0/java/nio/MappedByteBuffer/Truncate.d:/tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer \
-J-Dtest.file=/tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer/Truncate.java \
-J-Dtest.src=/tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer \
-J-Dtest.src.path=/tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer \
-J-Dtest.classes=/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/classes/0/java/nio/MappedByteBuffer/Truncate.d \
-J-Dtest.class.path=/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/classes/0/java/nio/MappedByteBuffer/Truncate.d \
-J-Dtest.vm.opts='-ea -esa -Xmx512m -Xshareclasses:cacheDir=/tmp/sverma/jdk_custom -Xjit:count=0' \
-J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-Xshareclasses:cacheDir=/tmp/sverma/jdk_custom -J-Xjit:count=0' \
-J-Dtest.compiler.opts= \
-J-Dtest.java.opts= \
-J-Dtest.jdk=/tmp/sverma/docker_jdk11/openj9-openjdk-jdk11/build/linux-s390x-normal-server-release/images/jdk \
-J-Dcompile.jdk=/tmp/sverma/docker_jdk11/openj9-openjdk-jdk11/build/linux-s390x-normal-server-release/images/jdk \
-J-Dtest.timeout.factor=8.0 \
-J-Dtest.root=/tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk \
-J-Dtest.nativepath=/tmp/sverma/docker_jdk11/openj9-openjdk-jdk11/build/linux-s390x-normal-server-release/images/test/jdk/jtreg/native \
-d /tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/classes/0/java/nio/MappedByteBuffer/Truncate.d \
-sourcepath /tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer \
-classpath /tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer:/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/classes/0/java/nio/MappedByteBuffer/Truncate.d /tmp/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/java/nio/MappedByteBuffer/Truncate.java
ACTION: main -- Failed. Unexpected exit from test [exit code: 255]
REASON: User specified action: run main/othervm Truncate
TIME: 2.975 seconds
messages:
command: main Truncate
reason: User specified action: run main/othervm Truncate
Mode: othervm [/othervm specified]
elapsed time (seconds): 2.975
configuration:
STDOUT:
STDERR:
JVMCDRT000E Unable to locate JIT stack map - aborting VM
JVMCDRT001E Method: jdk/internal/misc/Unsafe.getByte(J)B (00000000089D9600)
JVMCDRT002E Failing PC: 000003FF6CB16DB3 (offset 000000000000003F), metaData = 000003FF6C58B0F8
21:35:09.363 0x8b1a900j9codertvm(j9ji.110 * ** ASSERTION FAILED ** at jswalk.c:538: ((0 ))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2020/05/14 21:35:09 - please wait.
JVMDUMP032I JVM requested System dump using '/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/core.20200514.213509.9962.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.9989.
JVMDUMP010I System dump written to /tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/core.20200514.213509.9962.0001.dmp
JVMDUMP032I JVM requested Java dump using '/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/javacore.20200514.213509.9962.0002.txt' in response to an event
JVMDUMP012E Error in Java dump: /tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/javacore.20200514.213509.9962.0002.txt
JVMDUMP032I JVM requested Snap dump using '/tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/Snap.20200514.213509.9962.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /tmp/sverma/openjdk-tests/TKG/test_output_15894920702771/jdk_custom_0/work/scratch/0/Snap.20200514.213509.9962.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".
Grinder on the latest nightly to see if this problem is still reproducible: https://ci.eclipse.org/openj9/job/Grinder/897
I am having some trouble reproducing the failure with any additional options, TR_Options: count=0
, Xjit:count=0,disableasynccompilation
, or when running with reduced test bucket.
I have tried running the tests via the TKG test infrastructure, running them manually using the cmd grinder uses and through jdk_custom
Bellow are the commands that I have used to try to reproduce the failure
/home/jenkins/sverma/jdk11_images_with_my_changes/jdk/bin/java \
-Dtest.vm.opts='-ea -esa -Xmx512m -Xjit:count=0,disableasynccompilation' \
-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m' \
-Dtest.compiler.opts= \
-Dtest.java.opts= \
-Dtest.jdk=/home/jenkins/sverma/jdk11_images_with_my_changes/jdk \
-Dcompile.jdk=/home/jenkins/sverma/jdk11_images_with_my_changes/jdk \
-Dtest.timeout.factor=8.0 \
-Dtest.nativepath=/home/jenkins/sverma/jdk11_images_with_my_changes/test/jdk/jtreg/native \
-Dtest.root=/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk \
-Dtest.name=sun/nio/cs/FindDecoderBugs.java \
-Dtest.file=/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/sun/nio/cs/FindDecoderBugs.java \
-Dtest.src=/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/sun/nio/cs \
-Dtest.src.path=/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/sun/nio/cs:/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/lib \
-Dtest.classes=$compiled_tests/sun/nio/cs/FindDecoderBugs.d \
-Dtest.class.path=$compiled_tests/sun/nio/cs/FindDecoderBugs.d:$compiled_tests/test/lib \
-Dtest.class.path.prefix=$compiled_tests/sun/nio/cs/FindDecoderBugs.d:/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/sun/nio/cs:$compiled_tests/test/lib \
-Dseed=-8710075718699692142 \
-classpath $compiled_tests/sun/nio/cs/FindDecoderBugs.d:/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/jdk/sun/nio/cs:$compiled_tests/test/lib:/home/jenkins/sverma/openjdk-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/sverma/openjdk-tests/test-results/openjdk/jtreg/lib/javatest.jar:/home/jenkins/sverma/openjdk-tests/test-results/openjdk/jtreg/lib/jtreg.jar \
FindDecoderBugs
"/home/jenkins/sverma/jdk11_images_with_my_changes/jdk/bin/java" -Xmx512m \
-jar "/home/jenkins/sverma/openjdk-tests/test-results/openjdk/jtreg/lib/jtreg.jar" \
-agentvm -a -ea -esa -v:fail,error,time,nopass -retain:fail,error,*.dmp,javacore.*,heapdump.*,*.trc \
-ignore:quiet -timeoutFactor:8 -xml:verify -concurrency:2 \
-nativepath:"/home/jenkins/sverma/jdk11_images_with_my_changes/test/jdk/jtreg/native" -vmoptions:"-Xmx512m " \
-w ""/home/jenkins/sverma/openjdk-tests/TKG/../TKG/test_output_15909610427730/jdk_nio_0"/work" \
-r "/home/jenkins/sverma/openjdk-tests/test-results/openjdk/report" \
-jdk:"/home/jenkins/sverma/jdk11_images_with_my_changes/jdk" \
-exclude:"/home/jenkins/sverma/openjdk-tests/TKG/../openjdk/openjdk-jdk/test/jdk/ProblemList.txt" \
-exclude:"/home/jenkins/sverma/openjdk-tests/test-results/openjdk/ProblemList_openjdk11-openj9.txt" \
"/home/jenkins/sverma/openjdk-tests/TKG/../openjdk/openjdk-jdk/test/jdk:jdk_nio";
Quick update:
Turns out that the location of the core dump file in test output was misleading. The core files were silently moved to test_output_###/jdk_nio_0/work/sun/nio/cs/FindEncoderBugs/core.20200604.125452.27092.0001.dmp
thanks to @smlambert for pointing it out
re: https://github.com/eclipse/openj9/issues/9009#issuecomment-646110055 yes based on how many times people ask about it, I suppose someone should really propose a fix to openjdk jtreg framework to print that its moved the files.
Quick update: I haven't made much progress as I have been working on other items.
https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/13/ rh7-390-4 - history shows the test has passed on this machine before sun/nio/cs/FindDecoderBugs.java.FindDecoderBugs | 19 sec | 1 sun/nio/cs/SurrogateGB18030Test.java.SurrogateGB18030Test | 0.32 sec | 1 sun/nio/cs/SurrogateTestEUCTW.java.SurrogateTestEUCTW | 67 ms | 1 sun/nio/cs/SurrogateTestHKSCS.java.SurrogateTestHKSCS | 0.14 sec | 1 sun/nio/cs/Test4206507.java.Test4206507 | 90 ms | 1 sun/nio/cs/TestConverterDroppedCharacters.java.TestConverterDroppedCharacters | 0.28 sec | 1 sun/nio/cs/TestCp834_SBCS.java.TestCp834_SBCS | 0.38 sec | 1 sun/nio/cs/TestEBCDICLineFeed.java.TestEBCDICLineFeed | 88 ms | 1 sun/nio/cs/TestEUC_TW.java.TestEUC_TW | 0.3 sec | 1 sun/nio/cs/TestGB18030.java.TestGB18030 | 88 ms | 1 sun/nio/cs/TestHKSCS.java.TestHKSCS | 98 ms | 1 sun/nio/cs/TestIBMBugs.java.TestIBMBugs | 0.1 sec | 1 sun/nio/cs/TestISO2022CNDecoder.java.TestISO2022CNDecoder | 0.13 sec | 1 sun/nio/cs/TestISO2022JP.java.TestISO2022JP | 0.13 sec | 1 sun/nio/cs/TestISO2022JPEncoder.java.TestISO2022JPEncoder | 79 ms | 1 sun/nio/cs/TestISO2022JPSubBytes.java.TestISO2022JPSubBytes | 89 ms | 1 sun/nio/cs/TestIllegalSJIS.java.TestIllegalSJIS | 84 ms | 1 sun/nio/cs/TestJIS0208Decoder.java.TestJIS0208Decoder | 85 ms | 1 sun/nio/cs/TestJIS0212Decoder.java.TestJIS0212Decoder | 72 ms | 1 sun/nio/cs/TestMS5022X.java.TestMS5022X | 97 ms | 1 sun/nio/cs/TestMiscEUC_JP.java.TestMiscEUC_JP | 85 ms | 1 sun/nio/cs/TestSJIS0213.java.TestSJIS0213 | 89 ms | 1 sun/nio/cs/TestSJIS0213_SM.java.TestSJIS0213_SM | 79 ms | 1
sun/nio/cs/TestCp93xSISO.java.TestCp93xSISO | 69 ms | 1
sun/nio/cs/TestIllegalISO2022Esc.java.TestIllegalISO2022Esc | 91 ms | 1