Mechanism for injecting vault-ca.crt

[cpick]

Very excited for this vault/k8s integration!

In our setup we have vault installed within its own, "vault", namespace. It has a cluster-readable ConfigMap ("cm/vault-ca") that contains ".data.ca.crt". Currently, any Deployements/Jobs that would like to fetch a secret from vault GET "cm/vault-ca" from the k8s API Server and then use that to to TLS validation when communicating with vault.

With the injector I see that there is a vault.hashicorp.com/tls-secret annotation that will mount a Secret containing TLS certs (and that mount point can be referenced by vault.hashicorp.com/ca-cert), but since our "vault-ca" ConfigMap isn't a Secret (and, more importantly, is in a different namespace) we can't successfully mount it. We could try to duplicate "cm/vault-ca"s contents into Secrets in each Namespace, but then we'd have to worry about keeping them in sync and about how to protect them from tampering.

Since we've already installed the injector alongside vault in the "vault" namespace and since it's already modifying Pods as they're created, it seems like it would be able to actually insert the correct CA into the vault-agent init containers/sidecars (much like it already does with the $VAULT_CONFIG environment variable).

[jasonodonnell]

Hi @cpick, I think it could be a nice feature if we allowed setting the CA as an environment variable as part of the injection instead of mounting a secret. We'll consider doing this in the new year! For now you'll probably need to duplicate the CA until we support this.

[eliavem]

Any updates on this?

[joemiller]

Interested in this feature as well. Might also be able to send a PR for it, time permitting.

@jasonodonnell Do you envision setting the contents of a PEM-encoded CA cert in the agent-injector's deployment.yaml and then injecting that as a file in the vault agent initContainer + sidecar?

perhaps something along the lines of:

# agent-injector-deployment.yaml:
...
          args:
            - agent-inject
            - 2>&1
          env:
            - name: AGENT_INJECT_VAULT_CACERT
              value: |
                 -----BEGIN CERTIFICATE-----
                 ...
                 -----END CERTIFICATE-----

or perhaps it would be consumed via a file, possibly a ConfigMap or SecretVolume mounted into the agent injector container:

# agent-injector-deployment.yaml:
...
          args:
            - agent-inject
            - 2>&1
          env:
            - name: AGENT_INJECT_VAULT_CACERT
              value: /ca-certs/vault.pem
          volumeMounts:
            - mountPath: /ca-certs
              name: vault-ca
        volumes:
          - name: vault-ca
            secret:
              secretName: vault-ca

Then in the initContainer and sideCar container:

• if AGENT_INJECT_VAULT_CACERT was set, follow the existing pattern of b64 encoding the config.json into the VAULT_CONFIG env var:
• b64 encode the cacert into VAULT_CACERT env var
• append(?) additional commands to https://github.com/hashicorp/vault-k8s/blob/38c3f0c8d97766167b9b9d5430342655a63acdb3/agent-inject/agent/container_sidecar.go#L18 to decode and write the cacert to /home/vault.ca.pem
• set VAULT_CACERT=/home/vault/ca.pem environment variable on the init and sidecar container's podSpecs

It would be nice if Vault itself would accept a VAULT_CACERT env var that contained the contents of the cert instead of a path to the cert. That would simplify the mechanics here.

[jasonodonnell]

@joemiller,

We already support mounting via annotation by taking a secret name and mounting/configuring. I think where that fails to be beautiful is the secret must be present in the namespace the request originates from. We can't simply just mount the secret to each request because we can't guarantee it'll be available.

What we really don't want to do is add code to the injector that manages K8s secrets (ie: copying/creating CA secrets in your target namespace, then mounting it). The injector is intended to be a simple, unix style tool that does one job well: mutates pod specs to include Vault containers.

What is much more scalable, as hinted above, is if the injector is configured with a CA which we can add to the container via environment variable. The question here is do we consider that to be secret/sensitive data which would be in plaintext in the pod metadata? If we do think it's sensitive, then it goes to my point above of being cautious about adding K8s secret management code into the injector.

Thoughts?

[joemiller]

@jasonodonnell I totally agree about the injector not adding/removing Secrets or ConfigMaps into other namespaces.

What is much more scalable, as hinted above, is if the injector is configured with a CA which we can add to the container via environment variable.

I think I am saying this as well. The best I understand from this description this is what I was proposing above. The injector would load a CA cert at startup and then inject it into the mutated Pods using an environment var.

Maybe a briefer description would help clarify:

The webhook injector:

• AGENT_INJECT_VAULT_CACERT - introduce a new environment var. If set, this should point to a .pem bundle mounted into the injector's container.

A mutated pod:

• initContainer and sidecar env var: VAULT_CACERT_ENCODED. It contains the base64-encoded value of the CA cert that the webhook injector loaded when it started.
• The agent's command would be augmented to decode the environment variable and store it in a file such as /home/vault/ca.crt
• set VAULT_CACERT=/home/vault/ca.crt env var on the initContainer + sidecar container's definition.

Basically the same pattern for how the /home/vault/config.json is injected into the initcontainer + sidecar via the VAULT_CONFIG env var and then decoded before starting vault agent: "echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json"

This would be much simplified if Vault itself would allow the VAULT_CAERT env var contain the value of a CA bundle instead of a path to a file.

. The question here is do we consider that to be secret/sensitive data which would be in plaintext in the pod metadata?

No. CA certs should not be considered secret/sensitive data. =) As an extra measure of caution we could document that users should not accidentally store the CA key in the CA cert bundle. I'm not sure how often this happens, I hope not very often.

[sjpotter]

I'll be honest, it seems without a story that is universal for ca cert injection, the vault/kubernetes integration story is only half baked.

users shouldn't have to specify the ca cert to use for vault injector to get it. Vault injector should know it, and inject it itself always. All a user should care about in the common case is is 1) the file to write the secret out as, 2) the secret to use 3) the template for the file's output. Anything else should generally be transparent to the end user (configurable perhaps, but able to also be ignored for a sane set of default).

I'm having a hard time imagining common use cases where users wouldn't want this. Instead of it being configured in one place (i.e.: how the injector itself is configured), this punts it to every single pod/template on your system to implement in its annotations correctly. That just seems fundamentally wrong.

In my case, as an operator author trying to integrate vault support into our operator so that our clients can store their secrets in vault instead of as plain kubernetes secrets, this makes life difficult as there isn't a standard way of injecting the required data into the individual pods so that they can access vault in a standardized way. In our case, this is further compounded by the fact that we can't just use the injector to inject the secret into files, as our custom resources can use user specified secrets which we will fetch out of vault, so have to build (relatively easily) a vault https client to get the data (replacing the kube client used to read k8s secret data). All the data needed to build that client should be provided by the vault injector (and presumambly would be the same data the sidecar would use with its own vault client).

perhaps I'm missing something and there is, but based on a number of tickets I've seen on this topic that are still open, a lot of people must be missing it.

[rbkaspr]

Just adding another data point to this discussion, this is definitely something we need in our environment. Handling the cert injection in the mutation webhook sounds like a nice clean way to address it, just don't know what priority this issue has.

For anyone else looking for a somewhat clean workaround to this issue, I've thrown together an HAProxy config that basically presents an HTTP endpoint that the injectors inside the cluster can use, and then HAProxy transparently upgrades the connection to SSL and forwards it to the external Vault server. The proxy runs inside the cluster with Istio transparently securing that part of the connection with SSL.

# Add whatever boilerplate you need
frontend localhost
  bind 0.0.0.0:8200
  default_backend vault

backend vault
  server vault <VAULT_ADDRESS>:<VAULT_PORT> ssl

[smurfralf]

Its quite hard to believe this common-sense enhancement hasn't been prioritized. By not providing a simple reliable method of setting the CA certificate for the injected container regardless of namespace you are promoting the use of the vault.hashicorp.com/tls-skip-verify: "true" annotation, something which should be discouraged. Using annotations for this purpose should not be considered as the normal case. I should be able to supply the CA "bundle" (base64 encoded PEM content) when I deploy the injector webhook.

Other providers of vault connectivity don't seem to have an issue providing this capability. For example see the caBundle argument to cert-manager's vault issuer.

[sebglon]

Cert-manager team work on a trust-manager project to deploy cabundle via a configmap. This configmap need to be used like the kube-root-ca.crt. But this require a MutatingWebhook to add the configmap on each pod and init-container.

For my point of view, the vault-agent-injector need a solution to inject custom configmap via a webhook

[tomhjp]

Fixed in #507.