Open VinothChinnadurai opened 4 years ago
Hi @VinothChinnadurai, the Vault Agent is blocking until it successfully authenticates and currently has no way of timing out. We are looking at adding a timeout configurable to Vault agent (in the Vault project) to make this possible.
@jasonodonnell you mean at init container level the vault agent is trying to authenticate right? we already initialized the vault server once with the initial token. do we need to redo it once again?
@VinothChinnadurai Did you unseal Vault when you brought it back up?
The init container that was injected into your pod will block until it can successfully connect with Vault.
@jasonodonnell No, I felt it is not a needed one as we already did and the cluster is fully functional. I just tried to replicate say some server went down and brought back will automatically work without this need of initialization again. Please clarify
@VinothChinnadurai Vault must be unsealed if it goes down (this is different from initialization, which only happens once). If you aren't using Vault auto-unseal, then Vault is not going to be unsealed.
what will be the result of "vault status " command ? also check the logs of the vault pods ?
@VinothChinnadurai Without auto-unseal, you can leverage k8s secrets to store unseal keys and the postStart handler of the vault's container lifecycle to execute unseal operation. But if your k8s secrets are not properly encrypted, that would be a security hole.
@itisevgeny do you have more information regarding using K8S secrets to allow auto-unseal? I am looking for a simple way to unseal when the pod gets rescheduled. But have not found one that is properly contained.
@webmutation You can store unseal key(s) in k8s secret(s). Then create patch
spec:
template:
spec:
volumes:
- name: auto-unseal
secret:
secretName: unseal-key
containers:
- name: vault
volumeMounts:
- name: auto-unseal
readOnly: true
mountPath: "/vault/config/unseal"
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "KEY=$(cat /vault/config/unseal/key); vault operator unseal $KEY;"]
And apply that patch for vault statefulset.
Hi @itisevgeny Thanks for the tip, however do you think this is a safe approach? Probably could have limited access to the pod... i will definitely consider it.
I am currently trying to create a day 2 vault service, where I would use transit secrets to unlock the other vaults, , however the main keeper of vaults (Vault1) would need also an unseal mechanism and i would like to not rely on cloud provider KMS systems.
@webmutation I would say that the approach is not completely safe because k8s secrets are plain text by default. But there are options to achieve some level of safety that would be applicable. E.g. move Vault1 to separate cluster or configure encryption of k8s secrets, ...
I did a setup of vault + consul HA setup using the following repos on my AWS. https://github.com/hashicorp/vault-helm https://github.com/hashicorp/consul-helm Steps followed:
Installed Vault helm install vault -f values-eks.yaml ./vault-helm/ values-eks.yaml contains
Above steps booted 3 vault pods and 3 consul pods along with 3 consul servers spread evenly across 3 nodes.
Initialised vault kubectl exec -it vault-0 -- vault operator init
Unseal kubectl exec -it vault-0 -- vault operator unseal <unsealkey from step 3>
Modified Vault svc type to loadbalancer and got an ELB URL
Installed vault binary in another machine and set these things
Enabled Kubernetes auth mode
Create a Policy & Role:
vault policy write app ./app-policy.hcl
vault write auth/kubernetes/role/myapp \ bound_service_account_names=app \ bound_service_account_namespaces=demo \ policies=app \ ttl=24h
vault secrets enable -path=secret/ kv vault kv put secret/helloworld username=foobaruser password=foobarbazpass
spec: template: metadata: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-status: "update" vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld" vault.hashicorp.com/role: "myapp"
NAME READY STATUS RESTARTS AGE
app-864b96c9b6-b7lxw 0/2 Init:0/1 0 41h