Do not store local service account token and CA to config.

[tsaarni]

Overview

Kubernetes 1.21 switched to new ID tokens which are now bound to a specific pod. When pod is deleted and recreated the old token will be invalidated and not accepted by Kubernetes API server anymore. Tokens will also expire and Kubernetes will renew them periodically. The new tokens cause major problem with Vault since currently, even when using local tokens, the token and CA certificate are read when the authentication method is configured and stored persistently to config.

Kubernetes authentication backend will stop working when:

• the pod where the token happened to be copied from restarts (=pod gets recreated)
• token expires

Vault will not recover automatically, it requires reconfiguration of the authentication method.

Design of Change

This change avoids storing local token and local CA certificate in config. Instead, they are loaded lazily when needed and kept in memory only. Token will also re-read periodically to facilitate token rotation and to avoid authentication failure due to expired token.

Related Issues/Pull Requests

Fixes #121 Updates #95

Tests

Acceptance tests pass. New tests added. Manual test on Kubernetes pass..

Backwards compatibility

I believe that previously GET /auth/kubernetes/config returned a copy of local CA file. This is not done anymore. The justification is that local file is not part of stored configuration. CA file is still returned if user explicitly configured it with POST /auth/kubernetes/config.

Contributor Checklist

• [ ] Add relevant docs to upstream Vault repository, or sufficient reasoning why docs won’t be added yet Issue link https://github.com/hashicorp/vault/issues/12855
• [X] Add output for any tests not ran in CI to the PR description (eg, acceptance tests)
• [ ] Backwards compatible: see details under heading "Backwards compatibility" above

[tsaarni]

@hsimon-hashicorp Kindly asking if someone could take a look at this PR? Currently Vault's Kubernetes service account authentication method is broken and it would be really great to get it working again!

[heatherezell]

Chatted with @tvoran about this briefly; this is a note to myself to circle back on this later.

[tomhjp]

Hi @tsaarni, thanks for the PR! Sorry for the delay reviewing, but we're definitely interested in this feature. I haven't reviewed in detail yet, but will try to do so this week.

[tsaarni]

BTW I think it could be nice if there was a test environment with "fake" Kubernetes API server, allowing more thorough testing without compiling this plugin into Vault and running that against actual Kubernetes, in cluster and outside.

[tsaarni]

I've realized that I should rework the lock handling a bit: I need to have write lock for backend, when loading local files and assigning them to variables in kubeAuthBackend.loadConfig().

[tsaarni]

I wrote my manual test procedures here https://gist.github.com/tsaarni/66b085e0244fe1b0635cc1f806b5c334

[zohebs341]

Hi All,

We are facing a similar issue with Vault in our production workloads & and creating downtime. Can someone assist me with the vault part? I mean resolution for this.

A few days back we upgraded our Azure AKS Cluster from 1.20.7 to 1.21.7 and updated "disable_iss_validation=true" in vault config. It worked for some time, but intermittently we are facing issues with application pods which are crashing with permission denied errors and in vault logs, I can see below errors.

After going through multiple issues/docs, what I understood is vault is not able to load/read dynamically the service account/JWT tokens of pods, as it's changing frequently/expiring every hour in K8s version 1.21.7. Please assist me on this Error From Vault pod logs:

[ERROR] auth.kubernetes.auth_kubernetes_34b3b46a: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token Error From Application Pod Logs:

Error making API request.

Code: 403. Errors:

• 1 error occurred:
• permission denied (retry attempt 7 after "16s")

Config File:

Config File:

aultWriteAuthKubeConfig() {

echo && echo "[INFO]:: Writing Kubernetes auth config..."

vault write auth/kubernetes/config \

token_reviewer_jwt="$(cat ${KUBERNETES_SERVICE_TOKEN})" \

kubernetes_host=${KUBERNETES_HOST} \

kubernetes_ca_cert=@${KUBERNETES_CA_CERT} \

disable_iss_validation=true

[krishnamohan987]

@zohebs341 upgrade vault to V1.9.3 or + , it will support for dynamically reloading short-lived tokens.

[zohebs341]

@krishnamohan987 Thank you Krishna. We will try upgrading version.

[OneideLuizSchneider]

@krishnamohan987 Hi, the helm chart is still on v1.9.2, does it take a while to update there?

.

[krishnamohan987]

@OneideLuizSchneider you can override image version in values yaml while you apply the chart. it works. downlaod the image ready and push to your registry.