Closed tsaarni closed 2 years ago
@tomhjp, @tvoran could you please take a look at this issue, it is related to https://github.com/hashicorp/vault/issues/11953. Thank you!
Also encounter this issue on below env:
After deleted vault pod manually, the vault-agent-init sidecar which uses Kube-Auth failed to login, the error msg in vault pod:
login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
The workaround is: re-write auth/kubernetes/config
, then vault-agent-init can login again.
Just comment here to vote for this issue and subscribe to wait for formal fix.
Aso encounter this issue on below env:
After deleted vault pod manually, the vault-agent-init sidecar which uses Kube-Auth failed to login.
Just comment here to vote the issue
Same issue.
Recreating Kubernetes Auth Method solves until next reload.
@tsaarni Do we need to upgrade vault version from v1.7.3 to v1.10 or any other latest version as a part of fix?
Appreciate if you can share with us the vault config as well. As in our case, we are using v1.7.3 and vault config will differ from version to version.
Now in k8s v1.21.7, default service account expiry is 3607seconds.
Appreciate your response and support.
Yup, this feature requires 1.9.3 (which at time of writing is the latest version) or later. If Vault 1.9.3 or later is running inside the cluster, you only need to provide the Kubernetes host argument in most cases, e.g.:
kubectl exec -it vault-0 -- vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Confirmed that moving to 1.9.4 from 1.9.2, and changing auth/kubernetes/config
to not specify a token_reviewer_jwt
any longer, resolved this issue for us.
Now for the helm chart to be updated with 1.9.4 :)
I believe I've found some issues with the current service account token handling.
Background
(1) Kubernetes issues service account tokens that expire. Beginning from Kubernetes 1.21 the default expiration time for the token is 1 year but it can be configured down to 1 hour.
(2) Kubernetes associates issued tokens with particular pod via claim
\"kubernetes.io\".pod.name
.Example id token from Kubernetes 1.22.0
If the token is used after pod has been deleted,
401 Unauthorized
response will be returned and following error is logged by the API server:The claim is checked by the code here.
Problem description
Kubernetes Auth Backend seems to store the token persistently when auth method is configured by POSTing to
/auth/kubernetes/config
.The stored token will stop working when:
Proposed fix
Kubernetes Auth Backend could use similar approach as other Kubernetes clients do: