FIPS : Getting CKR_KEY_TYPE_INCONSISTENT when running Spark 3.3.0 using IBM Semeru Runtime 11.0.16.1 on FIPS enabled cluster.

[sshuklao]

After passing -Dsemeru.fips=true to jvm process, getting CKR_KEY_TYPE_INCONSISTENT error when running Spark 3.3.0 using IBM Semeru Runtime 11.0.16.1 on FIPS enabled cluster.

semerufips: FIPS mode detected, loading properties
semerufips: Removing provider: security.provider.12=SunPKCS11
semerufips: Removing provider: security.provider.11=JdkSASL
semerufips: Removing provider: security.provider.10=JdkLDAP
semerufips: Removing provider: security.provider.7=SunSASL
semerufips: Removing provider: security.provider.8=XMLDSig
semerufips: Removing provider: security.provider.9=SunPCSC
semerufips: Removing provider: security.provider.1=SUN
semerufips: Removing provider: security.provider.2=SunRsaSign
semerufips: Removing provider: security.provider.3=SunEC
semerufips: Removing provider: security.provider.4=SunJSSE
semerufips: Removing provider: security.provider.5=SunJCE
semerufips: Removing provider: security.provider.6=SunJGSS
semerufips: FIPS mode properties loaded
semerufips: {jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024, include jdk.disabled.namedCurves, policy.provider=sun.security.provider.PolicyFile, policy.url.1=file:${java.home}/conf/security/java.policy, jdk.security.legacyAlgorithms=SHA1, RSA keySize < 2048, DSA keySize < 2048, securerandom.source=file:/dev/random, policy.url.2=file:${user.home}/.java.policy, jdk.disabled.namedCurves=secp112r1, secp112r2, secp128r1, secp128r2, secp160k1, secp160r1, secp160r2, secp192k1, secp192r1, secp224k1, secp224r1, secp256k1, sect113r1, sect113r2, sect131r1, sect131r2, sect163k1, sect163r1, sect163r2, sect193r1, sect193r2, sect233k1, sect233r1, sect239k1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, X9.62 c2tnb191v1, X9.62 c2tnb191v2, X9.62 c2tnb191v3, X9.62 c2tnb239v1, X9.62 c2tnb239v2, X9.62 c2tnb239v3, X9.62 c2tnb359v1, X9.62 c2tnb431r1, X9.62 prime192v2, X9.62 prime192v3, X9.62 prime239v1, X9.62 prime239v2, X9.62 prime239v3, brainpoolP256r1, brainpoolP320r1, brainpoolP384r1, brainpoolP512r1, crypto.policy=unlimited, jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224, include jdk.disabled.namedCurves, jceks.key.serialFilter=java.base/java.lang.Enum;java.base/java.security.KeyRep;java.base/java.security.KeyRep$Type;java.base/javax.crypto.spec.SecretKeySpec;!*, jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, include jdk.disabled.namedCurves, X25519, X448, SSLv3, TLSv1, TLSv1.1, TLS_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, policy.ignoreIdentityScope=false, login.configuration.provider=sun.security.provider.ConfigFile, keystore.type.compat=true, security.overridePropertiesFile=true, jdk.tls.legacyAlgorithms=K_NULL, C_NULL, M_NULL, DH_anon, ECDH_anon, RC4_128, RC4_40, DES_CBC, DES40_CBC, 3DES_EDE_CBC, jdk.sasl.disabledMechanisms=, jdk.security.caDistrustPolicies=SYMANTEC_TLS, sun.security.krb5.maxReferrals=5, jdk.tls.keyLimits=AES/GCM/NoPadding KeyUpdate 2^37, security.provider.1=SunPKCS11 ${java.home}/conf/security/nss.fips.cfg, security.provider.2=SUN, security.provider.3=SunEC, networkaddress.cache.negative.ttl=10, jdk.tls.alpnCharset=ISO_8859_1, security.provider.4=SunJSSE, ssl.KeyManagerFactory.algorithm=SunX509, jdk.xml.dsig.secureValidationPolicy=disallowAlg http://www.w3.org/TR/1999/REC-xslt-19991116,disallowAlg http://www.w3.org/2001/04/xmldsig-more#rsa-md5,disallowAlg http://www.w3.org/2001/04/xmldsig-more#hmac-md5,disallowAlg http://www.w3.org/2001/04/xmldsig-more#md5,maxTransforms 5,maxReferences 30,disallowReferenceUriSchemes file http https,minKeySize RSA 1024,minKeySize DSA 1024,minKeySize EC 224,noDuplicateIds,noRetrievalMethodLoops, securerandom.drbg.config=, sun.security.krb5.disableReferrals=false, ssl.TrustManagerFactory.algorithm=PKIX, keystore.type=PKCS11, policy.allowSystemProperty=true, jdk.io.permissionsUseCanonicalPath=false, securerandom.strongAlgorithms=NativePRNGBlocking:SUN,DRBG:SUN, policy.expandProperties=true, package.access=sun.misc.,sun.reflect., package.definition=sun.misc.,sun.reflect., krb5.kdc.bad.policy=tryLast}

Caused by: java.security.InvalidKeyException: init() failed
    at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.engineInit(P11Mac.java:208)
    at java.base/javax.crypto.Mac.chooseProvider(Mac.java:366)
    at java.base/javax.crypto.Mac.init(Mac.java:435)
    at com.ibm.stocator.thirdparty.cos.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)
    ... 34 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_KEY_TYPE_INCONSISTENT
    at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_SignInit(Native Method)
    at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.initialize(P11Mac.java:177)
    at jdk.crypto.cryptoki/sun.security.pkcs11.P11Mac.engineInit(P11Mac.java:206)
    ... 37 more

[sshuklao]

looks like this issue is reported here in OpenJ9 too https://bugs.openjdk.org/browse/JDK-8282538.

[pshipton]

@alon-sh fyi

[paulcheeseman]

Possibly related?

https://access.redhat.com/solutions/6778751

[sshuklao]

@alon-sh any update?

[taoliult]

@sshuklao

The error message “CKR_KEY_TYPE_INCONSISTENT”, saying the specified key is not the correct type of key to use with the specified mechanism. So, in your case, what is the key and Mac algorithm it trying to use?

And the following link is a blog and examples, about how to use the FIPS in Semeru, especially the keystore. And currently the FIPS mode only support the P11 keys which stored in the NSSDB. So, how the keystore is configured in Spark 3.3.0, for using the Semeru FIPS?

https://www.ibm.com/support/pages/node/6612693

[mstoodle]

A fix for this problem has, I believe, been merged into OpenJ9 yesterday (e.g. https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/622#event-7565079702), which means it should be delivered as part of Semeru Runtimes 8u352, 11.0.17, etc. probably later this month.

[sshuklao]

@mstoodle is there any release date confirmed yet?

[mstoodle]

Not a precise one: OpenJDK released the last set of fixes yesterday afternoon, so our release process can now start. Hopefully by the end of the month.

[sshuklao]

@mstoodle I am still not seeing any build available with this fix.

[mstoodle]

Hopefully today or tomorrow. We got final confirmation of CVE fixes today, so we are starting the process to publish the binaries today.

[mstoodle]

Open Edition binaries have been posted here: https://github.com/ibmruntimes/semeru11-binaries/releases/tag/jdk-11.0.17%2B8_openj9-0.35.0

The web site may take longer to update, so pasting the direct download link to get you moving.

[aprenaud]

Closing as fixed, please reopen if you still experience the problem.