search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

jdk_security3_1_FAILED sun/security/pkcs11/ec/ReadCertificates.java PKCS11Exception: CKR_DATA_LEN_RANGE #18562

Closed JasonFengJ9 closed 7 months ago

JasonFengJ9 commented 9 months ago

Failure link

From an internal build(rhel8s390x-svl-rt7-1):

openjdk version "17.0.10" 2024-01-16
IBM Semeru Runtime Open Edition 17.0.10.0-m1 (build 17.0.10+5)
Eclipse OpenJ9 VM 17.0.10.0-m1 (build v0.43.0-release-9b2a42f8f, JRE 17 Linux s390x-64-Bit Compressed References 20240116_585 (JIT enabled, AOT enabled)
OpenJ9   - 9b2a42f8f
OMR      - d810fcb98
JCL      - 10ccdceee3c based on jdk-17.0.10+5)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2023-12-01T22:24:35.119Z] variation: Mode650
[2023-12-01T22:24:35.119Z] JVM_OPTIONS:  -XX:-UseCompressedOops -Xverbosegclog 

[2023-12-01T22:38:23.468Z] TEST: sun/security/pkcs11/ec/ReadCertificates.java

[2023-12-01T22:38:23.471Z] STDERR:
[2023-12-01T22:38:23.471Z] WARNING: A terminally deprecated method in java.lang.System has been called
[2023-12-01T22:38:23.471Z] WARNING: System::setSecurityManager has been called by PKCS11Test (file:/home/jenkins/workspace/Test_openjdk17_j9_extended.openjdk_s390x_linux/aqa-tests/TKG/output_17014636881420/jdk_security3_1/work/classes/0/sun/security/pkcs11/ec/ReadCertificates.d/)
[2023-12-01T22:38:23.471Z] WARNING: Please consider reporting this to the maintainers of PKCS11Test
[2023-12-01T22:38:23.471Z] WARNING: System::setSecurityManager will be removed in a future release
[2023-12-01T22:38:23.471Z] java.security.ProviderException: cancel failed
[2023-12-01T22:38:23.471Z]  at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.cancelOperation(P11Signature.java:346)
[2023-12-01T22:38:23.471Z]  at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.reset(P11Signature.java:291)
[2023-12-01T22:38:23.471Z]  at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.engineVerify(P11Signature.java:776)
[2023-12-01T22:38:23.471Z]  at java.base/java.security.Signature$Delegate.engineVerify(Signature.java:1435)
[2023-12-01T22:38:23.471Z]  at java.base/java.security.Signature.verify(Signature.java:789)
[2023-12-01T22:38:23.471Z]  at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:445)
[2023-12-01T22:38:23.471Z]  at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:384)
[2023-12-01T22:38:23.471Z]  at ReadCertificates.main(ReadCertificates.java:176)
[2023-12-01T22:38:23.471Z]  at PKCS11Test.premain(PKCS11Test.java:174)
[2023-12-01T22:38:23.471Z]  at PKCS11Test.testNSS(PKCS11Test.java:550)
[2023-12-01T22:38:23.471Z]  at PKCS11Test.main(PKCS11Test.java:210)
[2023-12-01T22:38:23.471Z]  at ReadCertificates.main(ReadCertificates.java:76)
[2023-12-01T22:38:23.471Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2023-12-01T22:38:23.471Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[2023-12-01T22:38:23.471Z]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2023-12-01T22:38:23.471Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:574)
[2023-12-01T22:38:23.471Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2023-12-01T22:38:23.471Z]  at java.base/java.lang.Thread.run(Thread.java:857)
[2023-12-01T22:38:23.471Z] Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DATA_LEN_RANGE
[2023-12-01T22:38:23.471Z]  at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.cancelOperation(P11Signature.java:319)
[2023-12-01T22:38:23.471Z]  ... 17 more
[2023-12-01T22:38:23.471Z] 
[2023-12-01T22:38:23.471Z] JavaTest Message: Test threw exception: java.security.ProviderException: cancel failed

[2023-12-01T23:06:32.258Z] jdk_security3_1_FAILED

50x internal grinder - 24/50 failed -Xint 50x internal grinder - reproduced w/ -Xint

pshipton commented 9 months ago

@jasonkatonica can someone look at this pls.

KostasTsiounis commented 9 months ago

The problem occurs at the part of the test where it tries to verify certificates with incorrect signatures (https://github.com/KostasTsiounis/openj9-openjdk-jdk17/blob/a9eedbbd76a3b207d8b467a63733551482b23109/test/jdk/sun/security/pkcs11/ec/ReadCertificates.java#L167). Two certificates are selected randomly, making sure they are not the same or their key algorithms don't match. In any other case, an expected exception is thrown, which is caught and the test succeeds.

The failure happens because there is a finally block (https://github.com/KostasTsiounis/openj9-openjdk-jdk17/blob/a9eedbbd76a3b207d8b467a63733551482b23109/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L775) that tries to cancel the operation after the exception is thrown. In cases where certificate and signer both use secp curves and the certificate's curve length is greater than the signer's, the cancelOperation() method's attempt to do another verify (https://github.com/KostasTsiounis/openj9-openjdk-jdk17/blob/a9eedbbd76a3b207d8b467a63733551482b23109/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L319) causes NSS to reply with a different return code (CKR_DATA_LEN_RANGE) that is not expected (https://github.com/KostasTsiounis/openj9-openjdk-jdk17/blob/a9eedbbd76a3b207d8b467a63733551482b23109/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L340). Said code leads to an also unexpected ProviderException that is not caught (https://github.com/KostasTsiounis/openj9-openjdk-jdk17/blob/a9eedbbd76a3b207d8b467a63733551482b23109/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L346).

This issue is only happening in s390x. When I tried it in x86, it either succeeds or the return code is an expected one (CKR_SIGNATURE_LEN_RANGE) that is handled and the appropriate exception is thrown.

I think this might be an issue with the NSS library in s390x. There are 3 available solutions:

pshipton commented 9 months ago

If user's may be caught out by the difference, doing Change the SunPKCS11 code to account for CKR_DATA_LEN_RANGE return code too. seems like the way to proceed.

pshipton commented 8 months ago

Moving this forward as it's too late for 0.43.