search

adoptium / infrastructure

This repo contains all information about machine maintenance.
Apache License 2.0
86 stars 102 forks source link

`sun/security/pkcs11` tests failing on RHEL8 systems #2900

Open sxa opened 1 year ago

sxa commented 1 year ago

Please set the title to indicate the test name and machine name where known.

To make it easy for the infrastructure team to repeat and diagnose, please answer the following questions:

Any other details: Split out from #2030 as that issue is resolved See also https://github.com/adoptium/infrastructure/issues/2360 and https://github.com/adoptium/infrastructure/issues/1829 which are related to issues on this RHEL8 system.

sxa commented 1 year ago

Failures occurred on the s390x RHEL8 system too ... Will assign myself so I can discuss with the Red Hat internal team.

sxa commented 1 year ago

Upstream issue: https://bugs.openjdk.org/browse/JDK-8295343

sxa commented 1 year ago

@smlambert What would you propose as the next action here?

smlambert commented 1 year ago

Exclude the tests in our problem lists under https://bugs.openjdk.org/browse/JDK-8295343 (and no we currently do not support by OS sub-flavours and supporting such granular exclusions is currently not in plan, unless somehow jtreg already offers that feature), and try to see an expedited fix to JDK-8295343 to be able to re-enable the tests quickly.

sxa commented 1 year ago

See also https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_2/86/testReport/ and https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/82/testReport/ which ran on test-equinix_esxi-ubuntu2204-x64-1 and test-equinix_esxi-ubuntu2204-x64-2, suggesting that Ubuntu 22.04 may be subject to the same issues as they both failed the following in the jdk_security3 suites:

New grinders to determine problematic tests

RHEL8/x64 failures (JDK8):

On test-docker-ubuntu2204-x64-1 (JDK20):

On test-equinix_esxi-ubuntu2204-x64-1 (JDK8):

Ubuntu 20.04: https://ci.adoptium.net/job/Grinder/6905/ (Same three failures as Ubuntu 22.04)

RHEL8/x64 JDK20: https://ci.adoptium.net/job/Grinder/6909/console - PASSED

Ubuntu 22.04 x64 JDK20: All passed: https://ci.adoptium.net/job/Grinder/6904/

(I was assuming we'd have a nice simple list of things to exclude ... :-) )

On RHEL8/x64: JDK8 (14 failures) JDK11 (16 failures) JDK17 (12 failures) JDK19 (12 failures) JDK20 Nightly (0 failures!) (Note - failures can be /2 since each one is there twice for the _0 and _1 suites)

Test JDK8 JDK11 JDK17 JDK19 JDK20
pkcs11/Signature/ByteBuffers [*] ❌ ❌ ✅ ✅ ✅
pkcs11/Signature/ReinitSignature [*] ❌ ❌ ✅ ✅ ✅
pkcs11/Signature/TestRSAKeyLength [*] ❌ ❌ ❌ ❌ n/a
pkcs11/fips/TestTLS12 ❌ ✅ n/a n/a n/a
tools/autotest.sh ❌ ❌ n/a n/a n/a
pkcs11/rsa/TestKeyFactory [+] ❌ ❌ ❌ ❌ ✅
pkcs11/rsa/TestSignatures ❌ ❌ ❌ ❌ ✅
pkcs11/KeyStore/Basic.java ✅ ❌ ❌ ❌ n/a
pkcs11/KeyPair/TestKeyPairGenerator n/a ❌ ❌ ❌ n/a
tools/NssTest n/a n/a ❌ ❌ n/a
lib/cacerts/VerifyCACerts n/a n/a n/a ❌ n/a
X509TrustManager/Distrust.java n/a n/a n/a ❌ n/a

[*] - the top 3 in this table are the ones definitely related to the enforced minimum key lengths in RHEL8, and suggests a fix may have gone into to 17+ [+] - Note that while sun/security/pkcs11/rsa/TestKeyFactory.java.TestKeyFactory is a failure, sun/security/pkcs11/rsa/TestKeyFactory passed

sxa commented 3 weeks ago

Need to determine correct actions in our exclude list or upstream in order to help this to pass consistently and run the appropriate tests.

sxa commented 1 week ago

@judovana Can you confirm whether you've seen these on your systems, and if you've excluded them in line with the table above?

judovana commented 1 week ago

I had checked all our rhel8 jobs, and the tests you are reffering are run, and passing. More details sent offlist.

sxa commented 1 week ago

Some new tests (Since the ones in the table above were from about 18 months ago)

JDK OS arch suite Results
jdk8u RHEL8 x64 jdk_security3_0 1 failure sun/security/pkcs11/fips/TestTLS12.java.TestTLS12
jdk8u UBI9 x64 jdk_security3_0 7 failures (*)
jdk8u UBI9 x64 jdk_security3 Same 7 failures as above (each duplicated twice)
jdk8u UBI9 arm64 jdk_security3 1 failure - autotest.sh
jdk8u UBI8 arm64 jdk_security3 PASS
jdk8u UBI9 arm64 jdk_security3_0 1 failure keytool/autotest.sh
jdk8u CS9 arm64 jdk_security3 2 failures autotest.sh and https://ci.adoptium.net/job/Grinder/11566/testReport/javax_net_ssl_ServerName_SSLEngineExplorerMatchedSNI/java/SSLEngineExplorerMatchedSNI/ javax.net.ssl.SSLProtocolException: Input record too big: max = 16709 len = 42359. Running with 100 iterations SNI passed 100/100
jdk8u(RH) RHEL8 x64 jdk_security3 3 failures autotest.sh, standard.sh and TestTLS12
jdk8u(RH) UBI9 x64 jdk_security3 8 failures 5 Secmod , standard.sh autotest.shandLogin.sh`
jdk8u(Zulu) RHEL8 x64 jdk_security3_0 Three failures
jdk8u(Zulu) UBI9 x64 jdk_security3_0 10 failures
jdk8u(Zulu) UBI9 x64 jdk_security3_0 10 failures
jdk8u(Bish) RHEL8 x64 jdk_security3_0 5 failures
jdk8u(D'well) RHEL8 x64 jdk_security3_0 8 failures
jdk8u(Semeru) RHEL8 x64 jdk_security3_0 1 failure SNI test (failed on Temurin on CS9)
jdk8u(Semeru) UBI9 x64 jdk_security3_0 5 failures Three Secmod issues, Login.sh and autotest.sh
jdk11u RHEL8 x64 jdk_security3_0 PASS
jdk17u RHEL8 x64 jdk_security3_0 4 failures
jdk11u UBI9 x64 jdk_security3_0 PASS
jdk17u UBI9 x64 jdk_security3_0 4 failures - same as on RHEL8
jdk21u RHEL8 x64 jdk_security3_0 4 failures - same as jdk17u
jdk21u UBI9 x64 jdk_security3_0 4 failures - same as jdk17u

(*) Here are the seven failures on UBI9: sun/security/pkcs11/Provider/Login.sh.Login 1.4 sec 1 sun/security/pkcs11/Secmod/AddPrivateKey.java.AddPrivateKey 0.27 sec 1 sun/security/pkcs11/Secmod/GetPrivateKey.java.GetPrivateKey 0.26 sec 1 sun/security/pkcs11/Secmod/JksSetPrivateKey.java.JksSetPrivateKey 0.2 sec 1 sun/security/pkcs11/Secmod/LoadKeystore.java.LoadKeystore 0.21 sec 1 sun/security/pkcs11/Secmod/TrustAnchors.java.TrustAnchors 0.23 sec 1 sun/security/tools/keytool/autotest.sh.autotest 2.4 sec 2

Noting that when trying with the Red Hat jdk8u build there was one additional failure:

Zulu showed three failures on RHEL8:

And zulu had 10 on UBI9:

JDK17u/21u failures on RHEL8/UBI9:

sxa commented 1 week ago

Summary of the above (unless otherwise stated these comparisons are on jdk8u/x64)

sxa commented 6 days ago

Builds without our custom CA Certs bundle for feeding to the upstream parameters in Grinder:

Running on UBI9 with the jdk21u build above at https://ci.adoptium.net/job/Grinder/11651/console (May take a while since there are a lot of jobs queued up)

sxa commented 5 days ago

Running on UBI9 with the jdk21u build above at https://ci.adoptium.net/job/Grinder/11651/console (May take a while since there are a lot of jobs queued up)

Showed the same four errors so it seems unrelated our custom cacerts.

Running with Red Hat jdk21u on UBI9: https://ci.adoptium.net/job/Grinder/11717 - Failed the same four tests.