vault-agent container cannot read secrets from kubernetes secrets engine

fizzamahdi commented 1 year ago

Unlike the AWS and Database Secrets Engines where the "creds/:name" endpoints support GET requests, the Kubernetes Secrets Engine's "creds/:name" endpoint only supports POST requests. That type of request is currently not supported by the vault-agent container that gets injected into pods by the Vault Agent Injector.

Will the Vault Agent be able to support this in the future?

Current setup

We have two EKS clusters and there is an application in cluster A that needs access to the Kubernetes API in cluster B. A workload in Cluster A authenticatea to Vault via a kubernetes auth method for Cluster A and the Vault role has a policy attached to it that gives it the ability to read credentials from a kubernetes secrets engine for Cluster B.

Here are the specific:

Kubernetes role in Cluster B (test-role-list-pods)
Vault Kubernetes secrets engine for Cluster B (kubernetes/puas/us-east-1/maas-k8s/)
Vault Kubernetes secrets engine role tied to (k8s-secrets-engine-example)
Kubernetes service account in Cluster A (k8s-auth-example)
Vault Kubernetes auth method for Cluster A (kubernetes-puas-ana-k8s)
Vault Kubernetes auth method role for Cluster A (k8s-auth-example)

Vault policy attached to the Kubernetes auth role

path "kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example" {
capabilities = ["list", "read", "create", "update"]
}

Kubernetes Deployment vault annotations and service account in Cluster A (vault-k8s-secrets-engine-example):

  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "k8s-auth-example"
    vault.hashicorp.com/namespace: "PCAS/isbx"
    vault.hashicorp.com/secret-volume-path-token: "/secrets"
    vault.hashicorp.com/agent-inject-secret-token: "kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example"
    vault.hashicorp.com/agent-inject-template-token: |
      {{- with secret "kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example" -}}
      {{ index .Data.service_account_token }}
      {{- end }}
spec:
  serviceAccountName: k8s-auth-example

vault-agent-injector deployment image in Cluster A: vault-k8s:1.1.0
agent image that is injected into pods in Cluster A: vault:1.12.1

Error message

The vault-agent-init container in a vault-k8s-secrets-engine-example pod has the following error message:

2023-03-16T21:28:43.734Z [WARN] (view) vault.read(kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example): vault.read(kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example): Error making API request.

Namespace: PCAS/isbx/
URL: GET https://vault.infosec.ic1.statefarm:8200/v1/kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example
Code: 405. Errors:

* 1 error occurred:
    * unsupported operation

Additional info

We were using the vault agent to pull a long-lived static kubernetes service account token from a KV secrets engine, so we want to switch over to using the kubernetes secrets engine. However, since we are using EKS, we have an option of using AWS credentials to get a kubernetes token (i.e. aws eks get-token) as a workaround if needed.

tomhjp commented 1 year ago

The templating language supports POST, but it's implicit, similar to how curl will automatically switch to POST if you provide body data. I think this modification should make it work (filling in the appropriate k8s namespace that you want the creds to be created in):

     vault.hashicorp.com/agent-inject-template-token: |
       {{- with secret "kubernetes/puas/us-east-1/maas-k8s/creds/k8s-secrets-engine-example" kubernetes_namespace=foo -}}
       {{ index .Data.service_account_token }}
       {{- end }}

fizzamahdi commented 1 year ago

Thank you for the clarification! Just adding the body worked for me.

hashicorp / vault-k8s