Closed smurfralf closed 4 years ago
@smurfralf This scenario should work, provided pods can communicate with the external Vault cluster. You will need to set AGENT_INJECT_VAULT_ADDR
on the deployment, as you already noticed. I will document the manual installation process using the deploy
files in this repo.
I'm struggling with using the deploy manifests from this repo, which i'm attempting to use because the helm charts are currently not working as intended with regards to global.enable variable, among other templating issues.
FWIW - I was able to get it working by changing the following
~1) I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.~ 2) I changed AGENT_INJECT_VAULT_ADDR to my vault server url ~3) I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )~
I will report back here as I become less confused :)
Hi @ls-brentsmith,
I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.
That RBAC is not needed by the injector. The injector doesn't actually communicate with Vault at all. The rbac you listed is what Vault uses when authenticating service accounts in Kubernetes.
I changed AGENT_INJECT_VAULT_ADDR to my vault server url
This should really be the only thing you need to do.
I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )
The deploy examples in this repository use the vault namespace. You should only need to create a namespace vault
and run the following (assuming you configured the AGENT_INJECT_VAULT_ADDR
with your external Vault server:
kubectl create namespace vault
kubectl create -f ./deploy/injector-rbac.yaml --namespace=vault
kubectl create -f ./deploy/injector-service.yaml --namespace=vault
kubectl create -f ./deploy/injector-mutating-webhook.yaml --namespace=vault
kubectl create -f ./deploy/deployment.yaml --namespace=vault
kubectl get pods --namespace=vault
@jasonodonnell thanks for the quick response.
In the light of morning I saw why I stumbled there.
Follow up question, is it possible to inject a custom auth path for the vault-agents?
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!
Thank you for the quick response! I was just about to come comment the same thing.
Are there plans to add an annotation for custom auth paths? It'd be nice to have a choice between a configmap or just annotations.
Yes, I'll be adding one @buckner!
@jasonodonnell - back to the original topic of this issue...
Your comment WRT setting external URL for AGENT_INJECT_VAULT_ADDR was: "This should really be the only thing you need to do." I did it using name "vault-inj" for the helm install, and the injector is indeed functional. Yay!
But my k8s cluster also has a zombie pod vault-inj-0 showing
Readiness probe failed
because of course this vault server hasn't been initialized.
When I made a stab at setting helm value service.enabled: false
then that breaks the injector where logs show
2019-12-23T16:40:45.791Z [INFO] handler: Starting handler.. Listening on ":8080"... Updated certificate bundle received. Updating certs... 2019/12/23 16:40:46 http: TLS handshake error from 10.0.16.98:41948: no certificate available
My guess is that you have dependencies there in your automagic configuration of TLS?
At any rate, solving that isn't my point. What I asked for in the description would be nice:
... please document an example configuration that (allows configuration of an external vault server [preferably without changing the vault-helm template] and that) also disables installing vault as a service inside kubernetes.
@smurfralf There are some dependencies in the Vault Helm project between the projects, so some stuff will need to change over there for installing the injector standalone.
I'll document the process of configuring the injector with an external Vault service after the new year since HashiCorp is currently in a holiday shutdown.
@jasonodonnell I have a setup of external vault server with consul backend and placed a load balancer in front of vault server in an EKS environment. Now m app has been deployed in separate EKS cluster where I mentioned AGENT_INJECT_VAULT_ADDR as the above mentioned ELB(Port 80). Now my app pod becomes not ready after I injected the init container . STATUS becomes 'Init:0/1' . I think it is something related to token initialization. How can I mention that token here and retrieve the secrets inside the Pod?
@VinothChinnadurai Did you check the init container logs to see what's happening?
kubectl logs <name of pod> -c vault-agent-init
Yes @jasonodonnell . Please find it here `URL: PUT http://vault.demo.svc:8200/v1/auth/kubernetes/login Code: 400. Errors:
Not sure why it still hits some other service(vault.demo.svc) instead of ELB address
@VinothChinnadurai Double check the env is set correctly because that looks like the default https://github.com/hashicorp/vault-k8s/blob/611492d04b7bf2115e3b43e8d148cc643a47d10d/deploy/injector-deployment.yaml#L36
Next retrigger the injection by patching the pod. You'll need to set the status annotation to vault.hashicorp.com/agent-inject-status: update
to retrigger the injection (the config needs to be refreshed on the agent).
@jasonodonnell Happy New year! Thanks. It worked for me! But I find an issue with the syncing part where I have updated my secrets through vault API command
curl \ --header "X-Vault-Token: xxx" \ --request POST \ --data @payload.json \ http://myelbaddress:8200/v1/secret/helloworld
Payload.json has
{ "data": { "foo": "bar", "zip": "zap" } }
Post then in the GET command of API, I can receive the updated secrets. I tried to see the same by logging inside my app pod and verified the file /vault/secret/helloworld Where it still showed the older values. Kindly help me with what might be the cause.
@VinothChinnadurai Lower the TTL of your secret. Updates are based on TTL of the secret.
@jasonodonnell does this also work with istio service mesh?
@trx35479 We haven't done explicit testing with istio yet, so we can not verify if this will or will not work with it. If you happen to test this, please let us know what you find.
@VinothChinnadurai Double check the env is set correctly because that looks like the default
Next retrigger the injection by patching the pod. You'll need to set the status annotation to
vault.hashicorp.com/agent-inject-status: update
to retrigger the injection (the config needs to be refreshed on the agent).
========================================================== @jasonodonnell I have consul+vault installed on one Kubernetes cluster. On the other cluster, I am trying to deploy vault-k8s injector for webhook - https://github.com/hashicorp/vault-k8s.git
Init pod returns the following errors. how can it be fixed? Vault address has been changed to http://vlt.consulvault.172.31.101.63.xip.io
Both the clusters are in the same network and curl command returns the response.
I think I may have to pass the root token(or register secret with Vault) in order to authenticate.
Thank you
curl \
> -H "X-Vault-Token: token" \
> -X GET \
> http://vlt.consulvault.172.31.101.63.xip.io/v1/secret/helloworld
{"request_id":"***","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"password":"foobarbazpass","username":"foobaruser"},"wrap_info":null,"warnings":null,"auth":null}
==> Vault server started! Log data will stream in below:
2020-01-28T18:27:10.042Z [INFO] sink.file: creating file sink
2020-01-28T18:27:10.042Z [INFO] sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.3.1
2020-01-28T18:27:10.042Z [INFO] auth.handler: starting auth handler
2020-01-28T18:27:10.042Z [INFO] auth.handler: authenticating
2020-01-28T18:27:10.042Z [INFO] template.server: starting template server
2020/01/28 18:27:10.042917 [INFO] (runner) creating new runner (dry: false, once: false)
2020/01/28 18:27:10.043280 [INFO] (runner) creating watcher
2020-01-28T18:27:10.043Z [INFO] sink.server: starting sink server
2020-01-28T18:27:10.275Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:
* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=2.382848811
2020-01-28T20:21:34.792Z [INFO] auth.handler: authenticating
2020-01-28T20:21:34.806Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:
* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=1.805414534
2020-01-28T20:21:36.612Z [INFO] auth.handler: authenticating
2020-01-28T20:21:36.758Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=1.998397118
2020-01-28T20:21:38.757Z [INFO] auth.handler: authenticating
2020-01-28T20:21:38.761Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=2.780268301
2020-01-28T20:21:41.541Z [INFO] auth.handler: authenticating
@jasonodonnell @shlee322 I've noticed the code for setting a custom path for the Kubernetes Auth method has been committed to master. Any chance you guys are going to do a release of that soon?
Hi @stevegore just released in v0.2.0
. Hope that helps!
Wrote this for myself and my team. It was pieced together using the awesome HashiCorp docs. All the examples follow what's in hashicorp/vault-guides repo and Learning modules on Hashi site. Hope one of you find this helpful.
There are two distinct parts of configuring auto-injection of secrets, Kubernetes & Vault. This will go through how-to configure both in order for auto-injection to work, as well as, a few custom options that should be considered as we deploy to production. This guide combines and refines two separate tutorials provided by Learn Vault for our specific use case.
$ cat sa-vault-agent-injector.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
$ kubectl apply --filename sa-vault-agent-injector.yaml
$ cat sa-vault-auth.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
$ kubectl apply --filename sa-vault-auth.yml
$ cat clusterrole-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: vault-agent-injector-clusterrole
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["mutatingwebhookconfigurations"]
verbs:
- "get"
- "list"
- "watch"
- "patch"
$ kubectl apply --filename clusterrole-vault-agent-injector.yaml
$ cat clusterrolebinding-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-agent-injector-binding
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
name: vault-agent-injector
namespace: default
$ kubectl apply --filename clusterrolebinding-vault-agent-injector.yaml
$ cat clusterrolebinding-vault-auth.yml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
$ kubectl apply --filename clusterrolebinding-vault-auth.yml
$ cat svc-vault-agent-injector.yaml
---
# Source: vault/templates/injector-service.yaml
apiVersion: v1
kind: Service
metadata:
name: vault-agent-injector-svc
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
spec:
ports:
- port: 443
targetPort: 8080
selector:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
$ kubectl apply --filename svc-vault-agent-injector.yaml
$ cat mutatingwebhookconfigurations-vault-agent-injector.yaml
---
# Source: vault/templates/injector-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: vault-agent-injector-cfg
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
webhooks:
- name: vault.hashicorp.com
clientConfig:
service:
name: vault-agent-injector-svc
namespace: default
path: "/mutate"
caBundle:
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
$ kubectl apply --filename mutatingwebhookconfigurations-vault-agent-injector.yaml
As per Vault documentation, “By default, the Vault Agent Injector will process all namespaces in Kubernetes except the system namespaces kube-system and kube-public. To limit what namespaces the injector can work in a namespace selector can be defined to match labels attached to namespaces.”
namespaceSelector is the selector for restricting the webhook to only specific namespaces. This should be set to a multiline string. For more information see https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
$ cat deployment-vault-auto-injector.yaml
---
# Source: vault/templates/injector-deployment.yaml
# Deployment for the injector
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
template:
metadata:
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
component: webhook
spec:
serviceAccountName: "vault-agent-injector"
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
containers:
- name: sidecar-injector
image: "hashicorp/vault-k8s:0.2.0"
imagePullPolicy: "IfNotPresent"
env:
- name: AGENT_INJECT_LISTEN
value: ":8080"
- name: AGENT_INJECT_LOG_LEVEL
value: info
- name: AGENT_INJECT_VAULT_ADDR
value: https://<VAULT_ADDR>:8200
- name: AGENT_INJECT_VAULT_IMAGE
value: "vault:1.3.2"
- name: AGENT_INJECT_TLS_AUTO
value: vault-agent-injector-cfg
- name: AGENT_INJECT_TLS_AUTO_HOSTS
value: vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc
args:
- agent-inject
- 2>&1
livenessProbe:
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /health/ready
port: 8080
scheme: HTTPS
failureThreshold: 2
initialDelaySeconds: 2
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 5
$ kubectl apply --filename deployment-vault-auto-injector.yaml
Note the value for AGENT_INJECT_VAULT_ADDR
this will be unique for each Vault cluster. Also take notice of AGENT_INJECT_VAULT_IMAGE
, this image must be present locally if restricting pods from only pulling images from a specific set of container repositories, such as GCR.
$ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
# determine Kubernetes master IP address (no https://) via `kubectl cluster-info`
$ export K8S_HOST=<K8S_MASTER_IP>
# set VAULT_TOKEN & VAULT_ADDR before next steps
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="https://$K8S_HOST:443" \
kubernetes_ca_cert="$SA_CA_CRT"
VAULT_TOKEN
and VAULT_ADDR
environment variables must be set before trying to communicate with the Vault server.
Be aware that the vault-auth service account is associated with the authentication method. This is the service account created explicitly for the Kubernetes authentication. It is not to be associated with any other resource or action. As such, should be protected accordingly.
If access to a Vault server is restricted by firewall rules and the Kubernetes cluster is not on the same internal network as the Vault server, it is critical a path is carved from the Kubernetes cluster to the Vault server, IE. whitelisting IP address of egress traffic from cluster in Vault’s firewall rule.
$ vault secrets enable -path=internal kv-v2
$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
$ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
capabilities = ["read"]
}
EOH
$ vault write auth/kubernetes/role/internal-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=internal-app \
ttl=24h
$ cat sa-internal-app.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: internal-app
$ kubectl apply --filename sa-internal-app.yml
$ cat deployment-01-orgchart.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orgchart
labels:
app: vault-agent-injector-demo
spec:
selector:
matchLabels:
app: vault-agent-injector-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-injector-demo
spec:
serviceAccountName: internal-app
containers:
- name: orgchart
image: jweissig/app:0.0.1
kubectl apply --filename deployment-01-orgchart.yml
At this point the pod does not have access to the secret, as there are no Vault specific annotations within the deployment, yet.
$ cat deployment-02-inject-secrets.yml
---
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
$ kubectl patch deployment orgchart --patch "$(cat deployment-02-inject-secrets.yml)"
$ kubectl exec orgchart-<pod-id> --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]
Now, there should be annotations associated with the pod. This will create the init and sidecar containers associated with the secret auto-injection. Secrets will be stored under /vault/secret. Worth noting, is the vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config" annotation. The key dictates the name of the file to be database-config.txt with the value being the path within Vault’s KV store: internal/data/database/config.
$ cat deployment-03-inject-secrets-as-template.yml
---
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret "internal/data/database/config" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}
$ kubectl patch deployment orgchart --patch "$(cat deployment-03-inject-secrets-as-template.yml)"
$ kubectl exec -it orgchart-<pod-id> -c orgchart -- cat /vault/secrets/database-config.txt
postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard
There are multiple ways to configure how the secret is formatted, using the annotation template key vault.hashicorp.com/agent-inject-template-database-config.txt. For a complete list of annotation options, see the Vault Project links in the Attribution section below.
$ cat deployment-04-payrole.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payrole
labels:
app: vault-agent-injector-demo
spec:
selector:
matchLabels:
app: vault-agent-injector-demo
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/role: "internal-app"
vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret "internal/data/database/config" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}
labels:
app: vault-agent-injector-demo
spec:
serviceAccountName: internal-app
containers:
- name: payrole
image: jweissig/app:0.0.1
$ kubectl apply --filename deployment-04-payrole.yml
Secrets are bound to the service account. The service account for a deployment is specified in the spec.template.spec.serviceAccountName block. The corresponding Vault relation is specified in Vault’s authentication role
Secrets are bound to the namespace. Similar to the service account, the corresponding Vault relation is specified in Vault’s authentication role.
Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.
The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_pathHope that helps!
Hi,
I found an annotations option that help to custom the auth_path that easier than using config map. You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint: https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like that https://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
Thanks @dcatalano-figure for the write up with the demonstrable code.
I recently delivered a HashiCorp Learn guide titled Integrate a Kubernetes Cluster with an External Vault.
In this guide, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets from this Vault, and configure an injector only deployment to inject secrets into the pods from this Vault.
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path Hope that helps!Hi,
I found an annotations option that help to custom the auth_path that easier than using config map. You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint:
https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like thathttps://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the
method
section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path Hope that helps!Hi,
I found an annotations option that help to custom the auth_path that easier than using config map. You can use the annotation option of vault like:
vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"
Config annotation like that, the vault agent will call to endpoint:
https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login
instead of default like thathttps://<vault_address>:8200/v1/auth/kubernetes/login
It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.
Hope this help!
Indeed, this is a new feature in 0.2.0
I see this issue is resolved but could someone guide me to the final documentation. We have an external vault gke cluster and multiple application gke clusters. I would like to see how we can use this tool to integrate our existing vault to inject secrets to all those clusters.
@sandeepkatthi99: https://learn.hashicorp.com/vault/getting-started-k8s/external-vault
@jasonodonnell , as per this documentation it is using a minikube and it is using http protocol to connect to the external vault sever. Mine is an external vault server with https protocol which uses a ca authorized certificate. in this case how to change the helm command to create injector pod with certificate options? Please help me. If i just replace http with https I am gtting below error.
auth.handler: authenticating 2020-03-26T10:03:01.452Z [ERROR] auth.handler: error authenticating: error="Put https://****/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.96206
I have built my vault cluster using https://codelabs.developers.google.com/codelabs/vault-on-gke/index.html?index=..%2F..cloud#18
Hi @sandeepkatthi99,
You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:
vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
This issue should be closed since the original description has been satisfied by vault-helm v0.4 with injector.externalVaultAddr
value.
Thanaks @jasonodonnell jasonodonnell I will try the same.
@burtlo Thanks for the learning guide.
I just wanted to let you know that in the Configure Kubernetes authentication section, the Kubernetes authentication
link pointing to https://www.vaultproject.io/docs/auth/kubernetes.html displays a "Page Not Found".
There's a bug in the Vault website generator causing a bunch links to break and is something we're working on fixing. https://www.vaultproject.io/docs/auth/kubernetes/ will work and sorry for the inconvenience!
@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?
@therealsamlin we're mounting the Kubernetes auth method at a different path for each cluster. Then set vault.hashicorp.com/auth-path
appropriately on the injector per cluster.
@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?
basically what @stevegore said. would like to add that auth path value must be prefixed with auth/
in where if the mount was named k8s-cluster-01 the value would be auth/k8s-cluster-01
In my case, I have vault server running on GKE cluster A and I have installed Vault Agent Injector service on cluster B by passing externalVaultAddr which is exposed over http. But after deploying sample application on B along with necessary annotations to inject vault secrets, I am getting below error in vault-agent-init container of my sample app
2020-08-26T08:24:59.554Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://35.225.127.203/v1/auth/kubernetes/login
Code: 403. Errors:
* permission denied" backoff=2.908150444
2020-08-26T08:25:02.463Z [INFO] auth.handler: authenticating
@jasonodonnell Any idea how to resolve this?
Hi, I am also facing the same issue. (I have 2 gke cluster setup , Cluster A vault is installed with internal LB, cluster B for application
Error writing data to auth/kubernetes/login: Error making API request.
URL: PUT http://gke_internal_LB_IP:8200/v1/auth/kubernetes/login Code: 403. Errors:
and logs [ERROR] auth.kubernetes.auth_kubernetes_c78cbc33: login unauthorized due to: Post "https://cluster_app_ip/apis/authentication.k8s.io/v1/tokenreviews": dial tcp cluster_app_ip:443: i/o timeout
Hello,
This question may have been answered earlier - not sure.
While performing the setup on EKS side , after executing all the 4 yaml files under deploy folder, i am able to create the agent injector POD.
However, when trying to perform the setup on the vault side, i am trying to obtain the JWT token as below , it complains of the missing secret token in secrets :
export SERVICEACCOUNT_JWT_TOKEN=$(kubectl get secret -o jsonpath="{.data.token}" $(kubectl get serviceaccount vault-injector -n vault -o jsonpath="{.secrets[0].name}") | base64 -d) Error from server (NotFound): secrets "vault-injector-token-p5jjp" not found
I verify that the secret does in exist with kubectl get secrets -n vault command
Does the secret need to be created manually ? Is there a yaml to execute while deploying the agent injector itself ?
Please let me know.
Hi @sandeepkatthi99,
You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:
vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>" vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
Hello, @jasonodonnell
please explain how to do it, I am new in Vault
@asl-cloud99 We are also having the similar use case (I have 2 gke cluster setup with same VPC, Cluster A vault is installed with internal LB, cluster B for application). Are you able to make it to work? If so, can you please share the steps?
I deployed the injector via the standard helm chart, with --set injector.externalVaultAddr
as describe in the Integrate a Kubernetes Cluster with an External Vault tutorial, and I'm seeing a TLS handshake error in the injector log whenever I deploy an annotated POD (the devwebapp-with-annotations from the tutorial).
[centos@ip-10-5-58-194 (vaulttest) ~]$ kubectl logs vault-agent-injector-5657ff569b-kdn8n -c sidecar-injector
2021-06-15T19:35:24.646Z [INFO] handler: Starting handler..
Listening on ":8080"...
2021-06-15T19:35:24.654Z [INFO] handler.auto-tls: Generated CA
2021-06-15T19:35:24.655Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-16T17:11:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-17T14:47:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T12:23:24.000Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T15:11:36.470Z [ERROR] handler: http: TLS handshake error from 127.0.0.1:56344: EOF
I've plenty of other TLS handshake issues reported with a bad certificate error but not this EOF variant.
A colleague "changed service mesh mtls strict policy to permissive" and that got rid of the TLS handshake error but the injector still wasn't injecting the init/agent containers. Then she disabled the istio sidecar from the vault-agent-injector POD, and it started working.
Hi Guys, I've tried all things shared here and still getting:
2021-06-20T21:38:24.892Z [INFO] auth.handler: authenticating
2021-06-20T21:39:24.893Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.87s
HELM: helm install vault hashicorp/vault --set "server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200,server.extraEnvironmentVars.VAULT_ADDR=http://vault:8200"
APP.YML:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-app
spec:
selector:
matchLabels:
app: vault-app
template:
metadata:
labels:
app: vault-app
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject-secret-hello.txt: secret/hello
vault.hashicorp.com/role: vault-app
vault.hashicorp.com/agent-pre-populate: "false"
vault.hashicorp.com/service: "http://vault:8200"
spec:
serviceAccountName: vault-app
containers:
- name: debian
image: debian:latest
command: [sleep, infinity]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-app
namespace: default
labels:
app: vault-app
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-app
namespace: default
labels:
app: vault-app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
namespace: default
name: vault-app
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-ingress
spec:
podSelector: {}
ingress:
- {}
policyTypes:
- Ingress
VAULT_0 conf:
vault kv put secret/hello foo=world
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# disable_local_ca_jwt=true \
# disable_iss_validation=true
vault policy write vault-app - <<'EOF'
path "secret/hello" {
capabilities = ["create", "read", "update", "delete", "list"]
}
EOF
vault write auth/kubernetes/role/vault-app \
bound_service_account_names=vault-app \
bound_service_account_namespaces=* \
policies=vault-app ttl=24h
Would you please help me to spot what is wrong here? I'd really really appreciate some help ... P.D: I've check all the rolebindings + services accounts + webmutations elated to vault and all of them seem to be correct ...
Hi, I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document. Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.
However, I am facing authentication issue as shown below
2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating: error= | Error making API request. |
---|---|
URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login | |
Code: 403. Errors: | |
* permission denied |
backoff=4m1.09s
I have enabled debug, here is the output of it
2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret \"secret/data/devwebapp/config\" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
Could someone please help in resolving this issue...
Hi, I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document. Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.
However, I am facing authentication issue as shown below
2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating: error= | Error making API request. | | URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login | Code: 403. Errors: | | * permission denied backoff=4m1.09s
I have enabled debug, here is the output of it
2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret "secret/data/devwebapp/config" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
Could someone please help in resolving this issue...
Suggest you check the auth Method of the Cluster from where you are trying to retrieve secret.
The scenario we want to support is to use a vault server which pre-exists the kubernetes cluster. We want the vault-k8s injector capability to talk to this vault server.
Is this scenario supported? I see that the injector command (vault-k8s/subcommand/injector/flags.go) takes a vault-address argument. And I see that deployment (vault-helm/templates/injector-deployment.yaml) sets the AGENT_INJECT_VAULT_ADDR env variable.
If the injector can be configured to use an external service, please document an example configuration that also disables installing vault as a service inside kubernetes.