search

adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
125 stars 304 forks source link

jdk_security3 test failed with wrong checksum and No trusted certificate errors #2123

Open LongyuZhang opened 3 years ago

LongyuZhang commented 3 years ago

jdk_security3 test got two sub-tests failure for both hotspot and openj9, with all jdk versions:

JavaTest Message: Test threw exception: java.lang.Exception JavaTest Message: shutting down test

- Distrust test failed with the error of `No trusted certificate found`, with detailed infomation:

Execution failed: `main' threw exception: java.lang.Exception: Unexpected exception: sun.security.validator.ValidatorException: No trusted certificate found. Testing verisignclass3g3ca java.lang.Exception: Unexpected exception: sun.security.validator.ValidatorException: No trusted certificate found at Distrust.testTM(Distrust.java:190) at Distrust.main(Distrust.java:114) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127) at java.base/java.lang.Thread.run(Thread.java:834)

JavaTest Message: Test threw exception: java.lang.Exception: Unexpected exception: sun.security.validator.ValidatorException: No trusted certificate found JavaTest Message: shutting down test


Related Issue: https://github.com/eclipse/openj9/issues/10757
smlambert commented 3 years ago

auto exclude test jdk_security3

sxa commented 3 years ago

@adamfarley The error message above looks like it's checking individual certificates instead of the overall cacerts file - based on the call we had is your understanding different (is there another test?) It is interesting if the certificates have ended up different, but possibly reasonable given that we use the Mozilla CA bundles (implemented here) although you can run a build without that by using the custom-cacerts parameter mentioned in https://github.com/AdoptOpenJDK/ci-jenkins-pipelines/blob/master/FAQ.md#how-do-i-build-more-quickly

aahlenst commented 3 years ago 
Execution failed: `main' threw exception: java.lang.Exception: At least one cacert test failed.
ERROR: wrong checksum
50:45:B0:D4:2D:46:E2:91:D0:F2:C9:A7:CB:77:45:14:41:63:A2:EF:BD:A9:F4:01:96:F6:7B:42:B2:29:7F:71
Expected checksum
84:BB:36:9E:B0:07:A7:C5:7F:38:EC:36:82:5C:0F:46:C0:35:3B:B1:1F:06:C2:D0:47:B9:39:FA:87:64:E5:9D
ERROR: 138 entries, should be 95
ERROR: camerfirmachambersignca [jdk] is not in cacerts
ERROR: thawteprimaryrootcag2 [jdk] is not in cacerts
...
java.lang.Exception: At least one cacert test failed
    at VerifyCACerts.main(VerifyCACerts.java:365)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
    at java.base/java.lang.Thread.run(Thread.java:834)

JavaTest Message: Test threw exception: java.lang.Exception
JavaTest Message: shutting down test

This one is expected. We now have a different certificate store with a different number of certificates and different alias names (and the test works based on alias names). Same in the second test. Apart from that, we have removed the untrusted VeriSign certificates in January 2021.

adamfarley commented 3 years ago

@adamfarley The error message above looks like it's checking individual certificates instead of the overall cacerts file - based on the call we had is your understanding different (is there another test?) It is interesting if the certificates have ended up different, but possibly reasonable given that we use the Mozilla CA bundles (implemented here) although you can run a build without that by using the custom-cacerts parameter mentioned in https://github.com/AdoptOpenJDK/ci-jenkins-pipelines/blob/master/FAQ.md#how-do-i-build-more-quickly

It does both. The "ERROR: wrong checksum" message is the first sign something's gone wrong, and that appears to be the test checking a checksum for the entire cacert file. Code snippet below.

        System.out.println("cacerts file: " + CACERTS);
        md = MessageDigest.getInstance("SHA-256");

        byte[] data = Files.readAllBytes(Path.of(CACERTS));
        String checksum = toHexString(md.digest(data));
        if (!checksum.equals(CHECKSUM)) {
            atLeastOneFailed = true;
            System.err.println("ERROR: wrong checksum\n" + checksum);
            System.err.println("Expected checksum\n" + CHECKSUM);
        }

Edit: Andreas is correct, in that we'd expect a different hash due to the different cacerts file.

LongyuZhang commented 2 years ago

More test failure for jdk_security3 on hotspot:

xlinux failed tests: Grinder Link: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4651/testReport/

TEST: sun/security/pkcs11/fips/SunJSSEFIPSInit.java

11:34:56  java.lang.RuntimeException: 'SunJSSE.isFIPS(): true' missing from stdout 
11:34:56  
11:34:56    at jdk.test.lib.process.OutputAnalyzer.stdoutShouldContain(OutputAnalyzer.java:178)
11:34:56    at SunJSSEFIPSInit.main(SunJSSEFIPSInit.java:70)
11:34:56    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:34:56    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:34:56    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:34:56    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
11:34:56    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
11:34:56    at java.base/java.lang.Thread.run(Thread.java:829)
LongyuZhang commented 2 years ago

Windows failed tests: Grinder Link: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4656/testReport/

TEST: sun/security/ssl/SSLSessionImpl/NoInvalidateSocketException.java

12:30:54  java.lang.RuntimeException: One or more sessions were improperly invalidated.
12:30:54    at NoInvalidateSocketException.main(NoInvalidateSocketException.java:88)
12:30:54    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:30:54    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:30:54    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:30:54    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:30:54    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
12:30:54    at java.base/java.lang.Thread.run(Thread.java:829)
LongyuZhang commented 2 years ago

Aix failed tests: Grinder Link: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4653/testReport/

TEST: jdk/security/jarsigner/JarWithOneNonDisabledDigestAlg.java

11:54:34  java.lang.Exception: JarEntry testfile1 is not signed
11:54:34    at JarWithOneNonDisabledDigestAlg.checkThatJarIsSigned(JarWithOneNonDisabledDigestAlg.java:137)
11:54:34    at JarWithOneNonDisabledDigestAlg.main(JarWithOneNonDisabledDigestAlg.java:79)
11:54:34    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:54:34    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:54:34    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:54:34    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
11:54:34    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
11:54:34    at java.base/java.lang.Thread.run(Thread.java:829)

TEST: sun/security/util/RegisteredDomain/ParseNames.java

12:46:08  java.lang.Exception: Test failed.
12:46:08    at ParseNames.main(ParseNames.java:75)
12:46:08    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:46:08    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:46:08    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:46:08    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:46:08    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
12:46:08    at java.base/java.lang.Thread.run(Thread.java:829)

TEST: sun/security/util/RegisteredDomain/Versions.java

12:46:08  java.lang.RuntimeException: assertEquals: expected https://raw.githubusercontent.com/publicsuffix/list/3c213aab32b3c014f171b1673d4ce9b5cd72bf1c/public_suffix_list.dat to equal https://raw.githubusercontent.com/publicsuffix/list/cbbba1d234670453df9c930dfbf510c0474d4301/public_suffix_list.dat
12:46:08    at jdk.test.lib.Asserts.fail(Asserts.java:594)
12:46:08    at jdk.test.lib.Asserts.assertEquals(Asserts.java:205)
12:46:08    at jdk.test.lib.Asserts.assertEquals(Asserts.java:189)
12:46:08    at jdk.test.lib.Asserts.assertEQ(Asserts.java:166)
12:46:08    at Versions.main(Versions.java:63)
12:46:08    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:46:08    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:46:08    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:46:08    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:46:08    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
12:46:08    at java.base/java.lang.Thread.run(Thread.java:829)
LongyuZhang commented 2 years ago

For JDK 8:

TEST: sun/security/pkcs11/Secmod/TestNssDbSqlite.java (almost all platforms)

12:37:37  java.lang.NullPointerException
12:37:37    at TestNssDbSqlite.testRetrieveKeysFromKeystore(TestNssDbSqlite.java:78)
12:37:37    at TestNssDbSqlite.main(TestNssDbSqlite.java:69)
12:37:37    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:37:37    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:37:37    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:37:37    at java.lang.reflect.Method.invoke(Method.java:498)
12:37:37    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
12:37:37    at java.lang.Thread.run(Thread.java:825)

Aarch Linux: TEST: sun/security/tools/jarsigner/diffend.sh

12:38:00  ACTION: shell -- Failed. Execution failed: exit code 1
12:38:00  REASON: Assumed action based on file name: run shell diffend.sh 
12:38:00  TIME:   0.312 seconds
12:38:00  messages:
12:38:00  command: shell diffend.sh
12:38:00  reason: Assumed action based on file name: run shell diffend.sh 
12:38:00  elapsed time (seconds): 0.312
12:38:00  STDOUT:
12:38:00  jarsigner: unable to open jar file: diffend.jar
12:38:00  STDERR:
12:38:00  rm: cannot remove 'diffend.jar': No such file or directory

TEST: sun/security/tools/jarsigner/emptymanifest.sh

12:38:02  STDOUT:
12:38:02  jarsigner: unable to open jar file: em.jar
12:38:02  STDERR:
12:38:02  rm: cannot remove 'emptymanifest.jks': No such file or directory
12:38:02  rm: cannot remove 'em.jar': No such file or directory

Windows Test Failure: jdk_security3 has around 150 test failures, will keep it excluded so far. https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4723/testReport/

LongyuZhang commented 2 years ago

JDK 17&18 more failures: Linux Platform:

TEST: javax/net/ssl/ServerName/SSLEngineExplorerWithSrv.java Also fail for openj9

 TEST RESULT: Error. Program `/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960428ms).

TEST: sun/security/provider/SecureRandom/AbstractDrbg/SpecTest.java

11:39:50  TEST RESULT: Error. Program `/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960554ms).

TEST: sun/security/pkcs11/Secmod/AddTrustedCert.java

16:11:19  java.security.KeyStoreException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_READ_ONLY
16:11:19    at jdk.crypto.cryptoki/sun.security.pkcs11.P11KeyStore.engineSetEntry(P11KeyStore.java:1051)
16:11:19    at jdk.crypto.cryptoki/sun.security.pkcs11.P11KeyStore.engineSetCertificateEntry(P11KeyStore.java:517)
16:11:19    at java.base/java.security.KeyStore.setCertificateEntry(KeyStore.java:1256)
16:11:19    at AddTrustedCert.main(AddTrustedCert.java:106)
16:11:19    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
16:11:19    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
16:11:19    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
16:11:19    at java.base/java.lang.Thread.run(Thread.java:833)
16:11:19  Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_READ_ONLY
16:11:19    at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_CreateObject(Native Method)
16:11:19    at jdk.crypto.cryptoki/sun.security.pkcs11.P11KeyStore.storeCert(P11KeyStore.java:1569)
16:11:19    at jdk.crypto.cryptoki/sun.security.pkcs11.P11KeyStore.engineSetEntry(P11KeyStore.java:1047)
16:11:19    ... 7 more
sophia-guo commented 2 years ago

Note: Failures of sun/security/tools/jarsigner/diffend.sh sun/security/tools/jarsigner/emptymanifest.sh

are machine related. https://github.com/adoptium/infrastructure/issues/2623

sophia-guo commented 1 year ago

sun/security/ssl/X509TrustManagerImpl/Symantec/Distrust.java jdk20 https://ci.adoptium.net/job/Test_openjdk20_hs_extended.openjdk_aarch64_linux/24/testReport/junit/sun_security_ssl_X509TrustManagerImpl_Symantec_Distrust/java/Distrust/