Support re-using a single key when sharing LTPA

kabicin commented 1 month ago

What this PR does / why we need it?:

Does this PR introduce a user-facing change?

[ ] User guide
[ ] CHANGELOG.md

Which issue(s) this PR fixes:

Fixes #

leochr commented 1 month ago

@kabicin Thank you for the updates. I tested some scenarios and the LTPA server config with and without worked as expected, except for one scenario where the encryption key was changed:

created some instances with manageLTPA enabled and with and without managePasswordEncryption. That created the LTPA key, server config with password encrypted with default key and server config with password encrypted with user-provided key.
changed the key in wlp-password-encryption-key.
the encrypted LTPA password didn't change in the existing app, but the encryption key did. This created a mismatch and resulted in the error below.

To get around the error I deleted the secret with encrypted password and that generated a new one with the correct password. But the app pod didn't get the updated encrypted password, although the secret is mounted. I believe it's to do with mount type. Kubernetes only updates secret if it's fully mounted (not particular keys)

[9/13/24, 21:58:51:265 UTC] 0000003c com.ibm.websphere.crypto.PasswordUtil                        E CWWKS1856E: The password was not processed because an unknown password algorithm exception was reported.
com.ibm.websphere.crypto.UnsupportedCryptoAlgorithmException
        at com.ibm.ws.crypto.util.PasswordCipherUtil.aesDecipher(PasswordCipherUtil.java:269)
        at com.ibm.ws.crypto.util.PasswordCipherUtil.decipher(PasswordCipherUtil.java:202)
        at com.ibm.websphere.crypto.PasswordUtil.decode_password(PasswordUtil.java:638)
        at com.ibm.websphere.crypto.PasswordUtil.passwordDecode(PasswordUtil.java:437)
        at com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask.getKeyPasswordBytes(LTPAKeyCreateTask.java:52)
        at com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask.getPreparedLtpaKeyInfoManager(LTPAKeyCreateTask.java:59)
        at com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask.createRequiredCollaborators(LTPAKeyCreateTask.java:95)
        at com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask.run(LTPAKeyCreateTask.java:105)
        at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:280)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:839)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:1065)
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1145)
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:919)
        at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2260)
        at com.ibm.ws.crypto.util.PasswordCipherUtil.aesDecipher(PasswordCipherUtil.java:255)
        ... 11 more

[9/13/24, 21:58:51:300 UTC] 0000003c com.ibm.ws.logging.internal.impl.IncidentImpl                I FFDC1015I: An FFDC Incident has been created: "java.lang.NullPointerException com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask 123" at ffdc_24.09.13_21.58.51.0.log
[9/13/24, 21:58:51:303 UTC] 0000003c com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask    E CWWKS4106E: LTPA configuration error. Unable to create or read LTPA key file: /opt/ol/wlp/output/defaultServer/resources/liberty-operator/ltpa.keys

leochr commented 1 month ago

Another item, not specific to this PR, but we should address: Add the operator shortname as the prefix to the internal encryption key secret to avoid conflict between OLO and WLO. Users could have both installed in the same namespace. The source secret, provided by the user, can continue to retain its name, since we want that to be shared. wlp-password-encryption-key-internal --> olo-wlp-password-encryption-key-internal

leochr commented 1 month ago

Noticed the following error when the LTPA key secret was deleted. A new one was generated, but since the encrypted password wasn't updated, the following error occurred:

[9/13/24, 22:57:07:351 UTC] 0000003c com.ibm.ws.logging.internal.impl.IncidentImpl                I FFDC1015I: An FFDC Incident has been created: "javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. com.ibm.ws.security.token.ltpa.LTPAKeyInfoManager 267" at ffdc_24.09.13_22.57.07.0.log
[9/13/24, 22:57:07:392 UTC] 0000003c com.ibm.ws.logging.internal.impl.IncidentImpl                I FFDC1015I: An FFDC Incident has been created: "javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask 123" at ffdc_24.09.13_22.57.07.1.log
[9/13/24, 22:57:07:392 UTC] 0000003c com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask    E CWWKS4106E: LTPA configuration error. Unable to create or read LTPA key file: /opt/ol/wlp/output/defaultServer/resources/liberty-operator/ltpa.keys

leochr commented 1 month ago

@kabicin It'll be good to throw a warning (using StatusConditionTypeWarning) when managePasswordEncryption: true is set, but wlp-password-encryption-key is not present in the namespace. So that the status in the CR informs the user that their configuration for managePasswordEncryption is not processed and that they should create the secret. The warning should be cleared when the secret is created afterwards. Thank you

leochr commented 1 month ago

@kabicin Thank you for the updates. I tested with the new commits from today. Updated the password encryption key and it resulted in Secret being updated with a newly encoded password. Observed similar results for the scenario where the LTPA key was deleted. But those updates were only available to the application pods that are 'leaders'. The non-leader pods still had the old values.

Looking at the Pod creation timestamp, new pods for the non-leaders were created (as a result of updating the last-rotation annotation), but before the LTPA key/password Jobs completed. This resulted in the 'old' values being mounted into the new pods.

kabicin commented 1 month ago

@leochr The leader does CreateOrUpdate for the LTPA Liberty XML Secrets but due to the subPath in the volume mount, the Secret is not hot reloaded and requires a manual restart to propagate to the pod. It can be verified by deleting non-leader OpenLibertyApplications, after deletion will then reload the configuration properly. I can patch the reload in and we could revisit hot reloading the Secret in a future release.

A container using a Secret as a subPath volume mount will not receive Secret updates.

https://kubernetes.io/docs/concepts/storage/volumes/

leochr commented 1 month ago

@kabicin Regarding the scenario where managePasswordEncryption: true but secret is not provided, it'll be appropriate to error out using Reconciled status condition (as opposed to a warning).

For reference for Single Sign-on, the following error is thrown using Reconciled status condition when sso is enabled, but the relevant secret is not provided:

Secret for Single sign-on (SSO) was not found. Create a secret named "app-sso-olapp-sso" in namespace "olo1403" with the credentials for the login providers you selected in application image.: Secret "app-sso-olapp-sso" not found

leochr commented 1 month ago

Tested and validated the above scenarios, which previously resulted in errors, now works fine (using the WLO driver with similar codebase).

OpenLiberty / open-liberty-operator

Support re-using a single key when sharing LTPA #622