Mount configMap/secret as files, rather than env vars

[jwalcorn]

In the interest of security, we really shouldn't be passing all of our configuration - especially the sensitive stuff like passwords and API keys and such - as environment variables. This was originally done for simplicity, but we need to revisit this and adopt a solution that demonstrates best practices around secure configuration of a Kubernetes-based application. The solution needs to work for both the server.xml for each microservice (like where a datasource is defined, with its credentials), and for config/secret values needed in the Java code. Some potentially helpful links (haven't had time to fully think this through yet, so this is more a reminder to investigate this, than full prescriptive guidance):

https://openliberty.io/blog/2021/10/29/kube-secrets-210011.html#kubernetes_secrets

https://kubernetes.io/docs/tutorials/configuration/configure-java-microservice/configure-java-microservice/

Note we had a solution, back when we were in IBM, to use Hashicorp Vault for this, which worked fine (you can still see stanzas in each microservice's helm chart template about Vault). I had even written my own VaultConfigSource as an extension to MicroProfile Config, as I'd explained on slide 19 of my "Stock Trader Sample - Diagram.pptx":

But while that was great for when we were in IBM (which owns Hashicorp), it's not necessarily the best general answer that we want to force every Kyndryl customer to use, just to make their app more secure. Hence the idea of mounting the config map and secret as files, and getting the values from there, if we can make it work without too much disruption.

[jwalcorn]

I have no doubt that a solution can be found pretty easily for the Java code, where you can write logic to do stuff (or use libraries, like MicroProfile Config). The reason I haven't addressed this so far is we also need a solution that works for the server.xml for each microservice, where we are much more limited on our options. Perhaps an init container is needed, and put fancy logic in there? Could that be generalized, so I don't have to maintain a different one per microservice? Lots to think about here...

[jwalcorn]

Other options that come to mind:

1. Use a hyperscaler-specific secrets management tool, like AWS Secrets Manager or Azure Key Vault
2. Rewrite this operator in Go, instead of living with the limitations of Helm, and do fancy configuration logic in that Go-lang code
3. Move to having people encode or even encrypt sensitive values (and just-in-time decode/decrypt them in each microservice). So no plain-text passwords in the CR yaml (which also eliminates people with Kube access seeing all of your passwords via a kubectl get StockTrader cjot -o yaml). That's what WebSphere did with its security.xml, and Open Liberty has a flavor of that you can use in your server.xml.
4. We might could even have a mutating web hook (and/or admission controller?) that took a CR yaml with plain-text passwords and just-in-time encodes/encrypts them before applying them to the cluster
5. All of that being said, perhaps the "mount the secret as a file" is the simpler solution... ;)

[jwalcorn]

One last thought/reminder - though most of the Stock Trader microservices are Liberty-based, we do have one written for Quarkus (Account), and two written for Spring/Tomcat (CashAccount and one flavor of StockQuote). And I'd love to get at least one microservice written in a language other than Java someday (we kind of almost got there with tradr that was written in JavaScript/Node.js, but that was quickly thrown together for a demo and never maintained, and hasn't worked in years, sadly). Oh, and I have (the beginnings of) a SwiftUI-based client, that runs great as a native app on iPhone/iPad/MacOS. So solutions that work for any app server, or even any language, would be preferable.

And one more thought - we also need to better deal with SSL certs (some of which I just have burned in to the trust.p12 for each microservice...), so possibly when we address a proper way to do certificate management, we'd use the same approach/infrastructure for secrets management? And as I recall, service meshes have some stuff that can help there? Just a thought - those don't necessarily have to be tied together.