Vault should support reloading service account token from disk

[tsaarni]

I believe I've found some issues with the current service account token handling.

Background

(1) Kubernetes issues service account tokens that expire. Beginning from Kubernetes 1.21 the default expiration time for the token is 1 year but it can be configured down to 1 hour.

(2) Kubernetes associates issued tokens with particular pod via claim \"kubernetes.io\".pod.name.

Example id token from Kubernetes 1.22.0

{'aud': ['https://kubernetes.default.svc.cluster.local'],
 'exp': 1634052974,
 'iat': 1634049374,
 'iss': 'https://kubernetes.default.svc.cluster.local',
 'kubernetes.io': {'namespace': 'default',
                   'pod': {'name': 'myapp-674f595ff9-q6w4q',
                           'uid': '2ac5b2e8-f74d-4fef-b410-cc5bebf574b1'},
                   'serviceaccount': {'name': 'my-service-account',
                                      'uid': '2fa34b84-7014-4860-a84f-93707230c0cd'}},
 'nbf': 1634049374,
 'sub': 'system:serviceaccount:default:my-service-account'}

If the token is used after pod has been deleted, 401 Unauthorized response will be returned and following error is logged by the API server:

E1012 14:37:47.799799       1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

The claim is checked by the code here.

Problem description

Kubernetes Auth Backend seems to store the token persistently when auth method is configured by POSTing to /auth/kubernetes/config.

The stored token will stop working when:

1. Token expires within one year (or sooner if default has been changed).
2. When the pod (where the token originates from) is deleted.

Proposed fix

Kubernetes Auth Backend could use similar approach as other Kubernetes clients do:

• Load token from disk and do not store it persistently.
• Reload token when it is renewed.

[tsaarni]

@tomhjp, @tvoran could you please take a look at this issue, it is related to https://github.com/hashicorp/vault/issues/11953. Thank you!

[grantliu3]

Also encounter this issue on below env:

• OpenShift 4.8.13 (Kubernetes v1.21.1)
• Vault version: v1.8.2

After deleted vault pod manually, the vault-agent-init sidecar which uses Kube-Auth failed to login, the error msg in vault pod:

login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

The workaround is: re-write auth/kubernetes/config, then vault-agent-init can login again.

Just comment here to vote for this issue and subscribe to wait for formal fix.

[alfredo-gil]

Aso encounter this issue on below env:

• EKS 1.21 (kubernetes v1.21.2)
• Vault version: 1.7.2

After deleted vault pod manually, the vault-agent-init sidecar which uses Kube-Auth failed to login.

Just comment here to vote the issue

[up-to-you]

Same issue.

• k3s 1.22
• Vault 1.8.5

Recreating Kubernetes Auth Method solves until next reload.

[zohebs341]

@tsaarni Do we need to upgrade vault version from v1.7.3 to v1.10 or any other latest version as a part of fix?

Appreciate if you can share with us the vault config as well. As in our case, we are using v1.7.3 and vault config will differ from version to version.

Now in k8s v1.21.7, default service account expiry is 3607seconds.

Appreciate your response and support.

[tomhjp]

Yup, this feature requires 1.9.3 (which at time of writing is the latest version) or later. If Vault 1.9.3 or later is running inside the cluster, you only need to provide the Kubernetes host argument in most cases, e.g.:

kubectl exec -it vault-0 -- vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

[yorickdowne]

Confirmed that moving to 1.9.4 from 1.9.2, and changing auth/kubernetes/config to not specify a token_reviewer_jwt any longer, resolved this issue for us.

Now for the helm chart to be updated with 1.9.4 :)