search

adoptium / openj9-systemtest

Long running J9 tests
Other
5 stars 37 forks source link

jdk20 MiniMix bigdecimal #148

Closed pshipton closed 1 year ago

pshipton commented 1 year ago

The following tests failed on OpenJ9 jdk20. Seems to me we had a similar issue in jdk19 and the tests were removed.

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_mac_Nightly/1/tapResults/

LT  testFailure: testItem_0839(net.adoptopenjdk.test.bigdecimal.TestSuite057): expected:<-9[E+18]> but was:<-9[000000000000000000]>
LT  org.junit.ComparisonFailure: expected:<-9[E+18]> but was:<-9[000000000000000000]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite057.testItem_0839(TestSuite057.java:5778)

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_linux_Nightly/1/tapResults/

LT  testFailure: testItem_0159(net.adoptopenjdk.test.bigdecimal.TestSuite019): expected:<-4[E+1]> but was:<-4[0]>
LT  org.junit.ComparisonFailure: expected:<-4[E+1]> but was:<-4[0]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite019.testItem_0159(TestSuite019.java:1064)

LT  testFailure: testItem_0113(net.adoptopenjdk.test.bigdecimal.TestSuite048): expected:<3[.2E+2]> but was:<3[20]>
LT  org.junit.ComparisonFailure: expected:<3[.2E+2]> but was:<3[20]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite048.testItem_0113(TestSuite048.java:763)

LT  testFailure: testItem_0205(net.adoptopenjdk.test.bigdecimal.TestSuite048): expected:<-3[E+2]> but was:<-3[00]>
LT  org.junit.ComparisonFailure: expected:<-3[E+2]> but was:<-3[00]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite048.testItem_0205(TestSuite048.java:1429)

LT  testFailure: testItem_0282(net.adoptopenjdk.test.bigdecimal.TestSuite024): expected:<2[E+308]> but was:<2[00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]>
LT  org.junit.ComparisonFailure: expected:<2[E+308]> but was:<2[00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite024.testItem_0282(TestSuite024.java:2018)

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_ppc64_aix_Nightly/1/tapResults/

LT  testFailure: testItem_0328(net.adoptopenjdk.test.bigdecimal.TestSuite033): expected:<6[E+2]> but was:<6[00]>
LT  org.junit.ComparisonFailure: expected:<6[E+2]> but was:<6[00]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite033.testItem_0328(TestSuite033.java:2373)

LT  testFailure: testItem_0017(net.adoptopenjdk.test.bigdecimal.TestSuite015): expected:<0[E+2147483646]> but was:<0[]>
LT  org.junit.ComparisonFailure: expected:<0[E+2147483646]> but was:<0[]>
LT      at org.junit.Assert.assertEquals(Assert.java:115)
LT      at org.junit.Assert.assertEquals(Assert.java:144)
LT      at net.adoptopenjdk.test.bigdecimal.TestSuite015.testItem_0017(TestSuite015.java:150)

There are more in the extended.system testing for other platforms. https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK20/5/

pshipton commented 1 year ago

@llxia @Mesbah-Alam pls take a look.

Mesbah-Alam commented 1 year ago

Hi @pshipton ,

net.adoptopenjdk.test.bigdecimal.TestSuite057 is not excluded in JDK 19, it's actually working normally on 19 : https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_aarch64_mac_Nightly_testList_1/117/consoleFull. 

org.junit.ComparisonFailure: expected:<-9[E+18]> but was:<-9[000000000000000000]>

Is this failure on JDK20 aarch64 Linux indicating a real issue?

Similar failure is not seen on JDK20 AIX : https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_ppc64_aix_Nightly/1/tapResults/.

pshipton commented 1 year ago

If you look at the build https://openj9-jenkins.osuosl.org/job/Pipeline-Build-Test-JDK20/5/ you can see that different platforms are failing different tests. Can you pls try the tests on Adoptium and see if they similarly fail. OpenJ9 didn't change any related behavior for jdk20 afaik.

pshipton commented 1 year ago

I tried some grinders on the latest and the failures are still occurring with OpenJ9. https://openj9-jenkins.osuosl.org/job/Grinder/2087/ https://openj9-jenkins.osuosl.org/job/Grinder/2088/

pshipton commented 1 year ago

Running on an Adoptium jdk20 build also fails. https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2089/

17:06:05  openjdk version "20-beta" 2023-03-21
17:06:05  OpenJDK Runtime Environment Temurin-20+36-202303170331 (build 20-beta+36-202303170331)
17:06:05  OpenJDK 64-Bit Server VM Temurin-20+36-202303170331 (build 20-beta+36-202303170331, mixed mode, sharing)
Mesbah-Alam commented 1 year ago

From the readme: "The Math suite contains simple tests which are designed to exercise the Math apis".

Are these math APIs supposed to generate different results on different platforms?

Looking at a failed test method on AIX in https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_ppc64_aix_Nightly/1/tapResults/ : 

 testItem_0328(net.adoptopenjdk.test.bigdecimal.TestSuite033): expected:<6[E+2]> but was:<6[00]>

The same test passes on Aarch64 Linux in Adoptiom JDK20 : 

17:09:57  LT  21:09:56.909 -   33 JUnit[net.adoptopenjdk.test.bigdecimal.TestSuite033]  Weighting=1

Similarly, it passes on Aarch64 Linux on OpenJ9 : https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_linux_Nightly/1/tapResults/.

pshipton commented 1 year ago

Even on the same platform, different tests fail between OpenJ9 and Adoptium, and different tests fail from run to run. There is something else going on. Perhaps a random sub-set of tests is being run instead of a consistent set.

I looked at all of the particular failures below, and they all have checks for JavaSpecVersionChecker.isJDK13OrNewer() but the behavior has reverted to the pre Java 13 behavior. I'm guessing if you update isJDK13OrNewer() to return false for jdk20 then the tests will pass.

OpenJ9: https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_linux_Nightly/1/tapResults/ https://openj9-jenkins.osuosl.org/job/Grinder/2088/tapResults/ Adoptium: https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2089/tapResults/

smlambert commented 1 year ago

related: https://github.com/adoptium/aqa-systemtest/issues/482

Mesbah-Alam commented 1 year ago

I'm guessing if you update isJDK13OrNewer() to return false for jdk20 then the tests will pass.

Yes, confirming that that is indeed the case.

Also, following https://github.com/adoptium/aqa-systemtest/issues/482, updated the test to check for 12 < version < 20, instead of version > 12 and it passes in jdk20. Grinder links in following PR : https://github.com/adoptium/aqa-systemtest/pull/483.

Mesbah-Alam commented 1 year ago

The MiniMix_5m test is now passing on JDK 20 https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_extended.system_aarch64_mac_Nightly/8/tapResults/.

Can this issue be closed?