tspearconquest
commented
1 year ago Hi, fellow user here.
The post linked actually demonstrates (as issue 1) the problem when a secret is kept in an env var within the container itself. Meaning when someone sets ENV foo=bar in the Dockerfile.
When you run docker image inspect on your kubernetes nodes, you're inspecting the image which was downloaded to the CRI on the node.
In the case of AKV2K8S, the image itself on-disk is not modified. What happens is that the pod is mutated at admission to the cluster by the webhook to add an initContainer and modify the command of the workload container.
This initContainer contains a single binary and that binary is kept on a volume shared between the 2 containers. The binary runs as the command so it retrieves the secret from Keyvault, and puts the retrieved secret value in the environment in-memory without modifying the image stored on disk on the node. Then the original command is started by the /azure-keyvault-env binary, so that the environment is inherited.
This is documented here and is (IMHO) vastly more secure than storing a secret in a volume on-disk.
You can test this for yourself by connecting to your node with an injected workload and inspecting the image for your workload. You'd see that the ENV variables are the same as what is put in the kubernetes manifest some-secret-name@azurekeyvault. Then if you exec a shell into your pod, you can do cat /proc/<somePID>/environ and see that the in-memory value is different.
I don't have a cluster with Azure Monitor to test Issue 2, but I believe the same would hold true there as well, so I definitely encourage you to test and report back with your findings!
Hope this clarifies a bit for you, and if there is something to fix, let's put heads together and come up with a fix once we have concrete data. :)
Is your feature request related to a problem? Please describe.
Your current offering is either using plain K8s secrets or injecting secrets as environment variables into the pod. I wish to add functionality to support injecting secrets into pods as a file through shared volumes. There are security reasons to add this functionality.
This also adds the possibility to update secrets without restarting pods, as the file can be monitored by the application itself which can reload the configuration in itself.
Describe the solution you'd like
Either create a new injector specifically for injecting secrets directly from a sidecar into the pod through a share volume in a file or add this functionality to the existing injector (although it should probably be renamed then).
Describe alternatives you've considered
HashiCorp Vault has its own injector, which has the same idea as yours. It takes from a vault and passes this to a pod, being different in how they do it (as they do it through the filesystem and you through environment variables).
This is however not an alternative, as this means people have to move all their Azure Key Vault secrets to HashiCorp Vault.
Additional context
I am available to work on this if this feature is approved.