Open wernerb opened 3 years ago
Hey @wernerb, thanks for the report.
I think this might be a problem with Kubernetes and lifecycle hooks, but not 100% sure. The token revocation is being done by a preStop
hook and I think because the container is crashing, that lifecycle hook never gets actitvated.
Speaking internally with my team about possibly enhancing Vault Agent to do this instead of relying on preStop
.
I have the same issue with the vault.hashicorp.com/agent-revoke-on-shutdown: "true" annotation. I checked the events on the pod and I can seea 403 error message in the "vault-agent" pod when the hook is triggered. If I try the same call using curl and the token provided at the path: /home/vault/.vault-token, it works correctly. Somehow, if the call to revoke the token is done using the cli, the proper token is not set to the vault cli. Any ideas? Many thanks!
Describe the bug A wrong vault template is used (wrong permissions), this results in the main container not running and crashing and restarting. While its wrong that the vault template was faulty, this wasn't caught at first. The pod restarted all night (3000 times) and created a lot of vault tokens. This can potentially swap vault.
The following annotation is enabled:
the vault token is able to be retrieved but a certain secret is unable to be rendered. Many tokens are created in quick succession swamping vault.
The agent gets a signal to be killed here:
but does not execute token revoke actions.
To Reproduce Steps to reproduce the behavior:
Deploy application annotated with
Request a secret for which you have no permission or does not exist:
Observe that the pod crashes, vault tokens are not revoked on shutdown of agent
See error (vault injector logs, vault-agent logs, etc.)
Application deployment:
Expected behavior The annotation is honored even if the leases/secrets generated are unavailable/permisison denied. As long as the token is retrieved it should be revoked on exit.
Environment