search

OpenLiberty / open-liberty

Open Liberty is a highly composable, fast to start, dynamic application server runtime environment
https://openliberty.io
Eclipse Public License 2.0
1.15k stars 592 forks source link

Unable to use NSS database for PKCS11 keystore #20148

Open benzim opened 2 years ago

benzim commented 2 years ago

Describe the bug
The keystore entry in our server.xml is

<keyStore id="defaultKeyStore" location="${env.NSS_CONFIG_FILE}" type="PKCS11" fileBased="false" provider="SunPKCS11" password="${env.WLP_KEYSTORE_PASS}"/>

During initialization, we are seeing this error

Cannot open keystore URL: /configmaps/operational-cache/nss-client.cfg; java.lang.reflect.InvocationTargetException
CWPKI0814E: An error while initializing hardware keystore [defaultKeyStore].  Check the hardware configuration /configmaps/operational-cache/nss-client.cfg file to be sure the attributes are set correctly.  Exception returned from the provider is Secmod directory /staging/operational-cache/keystore invalid, NSS already initialized with /etc/pki/nssdb.
CWPKI0809W: There is a failure loading the defaultKeyStore keystore. If an SSL configuration references the defaultKeyStore keystore, then the SSL configuration will fail to initialize.  
Exception initializing KeyStore; java.lang.reflect.InvocationTargetException 
          java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore.getProviderLegacy(WSPKCSInKeyStore.java:262)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore.createPKCS11Provider(WSPKCSInKeyStore.java:251)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore.access$000(WSPKCSInKeyStore.java:49)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore$1.run(WSPKCSInKeyStore.java:218)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore$1.run(WSPKCSInKeyStore.java:215)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore.initializePKCS11ImplProvider(WSPKCSInKeyStore.java:215)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStore.<init>(WSPKCSInKeyStore.java:96)
    at com.ibm.ws.ssl.core.WSPKCSInKeyStoreList.insert(WSPKCSInKeyStoreList.java:77)
    at com.ibm.ws.ssl.config.WSKeyStore$1.run(WSKeyStore.java:952)
    at com.ibm.ws.ssl.config.WSKeyStore$1.run(WSKeyStore.java:812)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.ibm.ws.ssl.config.WSKeyStore.obtainKeyStore(WSKeyStore.java:812)
    at com.ibm.ws.ssl.config.WSKeyStore.do_getKeyStore(WSKeyStore.java:772)
    at com.ibm.ws.ssl.config.WSKeyStore.getKeyStore(WSKeyStore.java:1048)
    at com.ibm.ws.ssl.config.WSKeyStore.getKeyStore(WSKeyStore.java:1022)
    at com.ibm.ws.ssl.config.WSKeyStore.initializeKeyStore(WSKeyStore.java:1167)
    at com.ibm.ws.ssl.config.WSKeyStore.<init>(WSKeyStore.java:330)
    at com.ibm.ws.ssl.internal.KeystoreConfig.updateKeystoreConfig(KeystoreConfig.java:93)
    at com.ibm.ws.ssl.internal.KeystoreConfigurationFactory.updated(KeystoreConfigurationFactory.java:106)
    at com.ibm.ws.config.admin.internal.ManagedServiceFactoryTracker$2.run(ManagedServiceFactoryTracker.java:267)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.ibm.ws.config.admin.internal.UpdateQueue$Queue.run(UpdateQueue.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.security.ProviderException: Secmod directory /staging/operational-cache/keystore invalid, NSS already initialized with /etc/pki/nssdb
    at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:212)
    at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:131)
    ... 35 more

Steps to Reproduce
Set NSS database as PKCS11 keystore

Expected behavior
I expect WLP to use the NSS database as the keystore without any errors.

Diagnostic information:

jpfloresibm commented 2 years ago

I still see the issue on WLP version 22.0.0.3 but what only when running inside of a kubernetes pod with FIPS enabled. This issue only appears to happen inside of a kubernetes pod and works as expected outside of it. The following configuration was used:

server.xml

 <keyStore id="defaultKeyStore" location="/usr/operational-cache/wlp/usr/servers/mdm-server/nss-client.cfg" type="PKCS11" fileBased="false" provider="SunPKCS11" password="${env.WLP_KEYSTORE_PASS}"/>

nss-client.cfg

name = nss-clients
nssLibraryDirectory = /lib64
nssSecmodDirectory = /tmp/fips/keystore
nssDbMode = readWrite
nssModule = fips
attributes=compatibility
showInfo=true

keystore directory

sh-4.4$ ls /tmp/fips/keystore
cert8.db  key3.db  secmod.db

environment variables

NSS_DEFAULT_DB_TYPE=dbm
NSS_CONFIG_FILE=/usr/operational-cache/wlp/usr/servers/mdm-server/nss-client.cfg

trace output from host machine

[5/11/22 9:17:20:771 PDT] 0000001c SystemOut                                                    O NSS modules: [NSS Internal FIPS PKCS #11 Module (FIPS, /lib64/libsoftokn3.so, slot 0)]
[5/11/22 9:17:20:773 PDT] 0000001c SystemOut                                                    O Information for provider SunPKCS11-nss-client
[5/11/22 9:17:20:773 PDT] 0000001c SystemOut                                                    O Library info:
[5/11/22 9:17:20:780 PDT] 0000001c SystemOut                                                    O   cryptokiVersion: 2.40
  manufacturerID: Mozilla Foundation              
  flags: 0
  libraryDescription: NSS Internal Crypto Services    
  libraryVersion: 3.67
[5/11/22 9:17:20:780 PDT] 0000001c SystemOut                                                    O All slots: 3
[5/11/22 9:17:20:780 PDT] 0000001c SystemOut                                                    O Slots with tokens: 3
[5/11/22 9:17:20:781 PDT] 0000001c SystemOut                                                    O Slot info for slot 3:
[5/11/22 9:17:20:781 PDT] 0000001c SystemOut                                                    O   slotDescription: NSS FIPS 140-2 User Private Key Services                        
  manufacturerID: Mozilla Foundation              
  flags: CKF_TOKEN_PRESENT
  hardwareVersion: 3.67
  firmwareVersion: 0.00
[5/11/22 9:17:20:789 PDT] 0000001c SystemOut                                                    O Token info for token in slot 3:
[5/11/22 9:17:20:789 PDT] 0000001c SystemOut                                                    O   label: NSS FIPS 140-2 Certificate DB   
  manufacturerID: Mozilla Foundation              
  model: NSS 3           
  serialNumber: 0000000000000000
[5/11/22 9:17:20:890 PDT] 0000001c com.ibm.ws.ssl.config.WSKeyStore                             I Successfully loaded default keystore: /usr/operational-cache/wlp/usr/servers/mdm-server/nss-client.cfg of type: PKCS11
[5/11/22 9:17:21:302 PDT] 00000013 com.ibm.ws.security.jaspi.AuthConfigFactoryWrapper           I CWWKS1655I: The default Java Authentication SPI for Containers (JASPIC) AuthConfigFactory class com.ibm.ws.security.jaspi.ProviderRegistry is being used because the Java security proper

trace output from pod container

[ERROR   ] CWPKI0814E: An error while initializing hardware keystore [defaultKeyStore].  Check the hardware configuration /usr/operational-cache/wlp/usr/servers/mdm-server/nss-client.cfg file to be sure the attributes are set correctly.  Exception returned from the provider is Secmod directory /staging/operational-cache/keystore invalid, NSS already initialized with sql:/etc/pki/nssdb.

Once this i noticed is that it appears as though the default java.security (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-2.el8_6.x86_64/jre/lib/security/java.security) properties are being used instead of what is defined in the liberty keystore properties.

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-2.el8_6.x86_64/jre/lib/security/java.security

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
#security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

#
# Security providers used when global crypto-policies are set to FIPS.
#
fips.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.fips.cfg
fips.provider.2=sun.security.provider.Sun
fips.provider.3=sun.security.ec.SunEC
fips.provider.4=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS-FIPS

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-2.el8_6.x86_64/jre/lib/security/nss-fips.cfg

name = NSS-FIPS
nssLibraryDirectory = /usr/lib64
nssSecmodDirectory = sql:/etc/pki/nssdb
nssDbMode = readOnly
nssModule = fips
jpfloresibm commented 2 years ago

Trace log for pod with error has been provided below:

trace.log

jensengelke commented 1 year ago

Not sure if it is still relevant, but https://openliberty.io/docs/latest/enable-fips.html suggests to use provider="SunPKCS11-NSS-FIPS", which works for me when running natively on RHEL 8.6 and using /etc/pki/nssdb