"403 Permission denied" on `auth/kubernetes/login`

[pat-s]

Describe the bug I am not sure if this is a VSO issue but solving this might be helpful for users of Vault and VSO.

I am trying desperately for multiple days now to get VSO (via helm chart) working in a new cluster (I had it working once in another one but cannot replicate the config).

To Reproduce Steps to reproduce the behavior:

1. Deploy Vault helm chart with serviceaccount.createSecret.true (to be able to use the token as token_reviewer_jwt
2. Configure auth/kubernetes/login

vault read auth/kubernetes/config
Key                                  Value
---                                  -----
disable_iss_validation               true
disable_local_ca_jwt                 false
issuer                               n/a
kubernetes_ca_cert                   -----BEGIN CERTIFICATE-----
$READACTED
-----END CERTIFICATE-----
kubernetes_host                      https://$REDACTED:6443
pem_keys                             []
token_reviewer_jwt_set               true
use_annotations_as_alias_metadata    false

I am applying the config via

vault write auth/kubernetes/config \
       token_reviewer_jwt="$(kubectl get secret -n vault vault-token -o jsonpath='{.data.token}' | base64 --decode)" \
       kubernetes_host="$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')" \
       kubernetes_ca_cert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"

3. Create a Vault policy to map the role to

vault policy write k3s - <<EOF
     path "kvv2/*" {
       capabilities = ["read"]
     }
EOF

4. Create a role for the kubernetes auth engine (NB: leaving aud out to make life easier)

vault write auth/kubernetes/role/k3s bound_service_account_names=* bound_service_account_namespaces=* policies=k3s aud="" ttl=24h

5. Define VaultAuth

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: vault
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: k3s
    serviceAccount: default

6. Define a VaultStaticSecret

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: foo
  namespace: vault
spec:
  type: kv-v2

  # mount path
  mount: kvv2

  # path of the secret
  path: foo/bar

  # dest k8s secret
  destination:
    name: bar
    create: true

  # static secret refresh interval
  refreshAfter: 30s

  # Name of the CRD to authenticate to Vault
  vaultAuthRef: static-auth

Expected behavior The secret is created by VSO and everything works.

Actual behavior

VSO (and the API) tell me the following:

manager {"level":"error","ts":"2024-07-08T12:25:48Z","msg":"Reconciler error","controller":"vaultstaticsecret",
"controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","Vault StaticSecret":
{"name":"foo","namespace":"vault"},"namespace":"vault","name":"foo",
"reconcileID":"dea264dc-514e-42e4-b8a5-24d1da5f7b50","error":"Error making API request.
\n\nURL: PUT http://vault.vault.svc.cluster.local:8200/v1/auth/kubernetes/login\nCode: 
403. Errors:\n\n* permission denied"} 

Environment

• Kubernetes version: 1.29
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): k3s
  • Other configuration options or runtime services (istio, etc.):
• vault-secrets-operator version: 0.7.1

Additional context

I read the docs about configuring kubernetes auth as well as followed the VSO tutorial.

For VSO, I've tried setting defaultVaultConnection.address to both the internal connection at http://vault.vault.svc.cluster.local:8200 and the external one at https://vault.mydomain.xyz. Same problem.

I verified that the SA has a token and has a Clusterrole assigned which allows system:auth-delegator to be able to act as a JWT reviewer.

Besides trying to use the Use the Vault client's JWT as the reviewer JWT I also tried Use local service account token as the reviewer JWT but without any difference.

I might be overlooking something obvious. Can someone point me to it or tell me what I am doing wrong?

[pat-s]

Got it working after multiple days now by using kubernetes_host="https://kubernetes.default.svc/" for the kubernetes auth config.

VSO is still operating through the public route via defaultVaultConnection.address: https://my-domain.org.

So in the end it was caused by a connection issue of Vault itself due to the kubernetes_host setting.

[harshalk91tg]

@pat-s Would you be able to share values file? I am struggling with same issue.

[pat-s]

@harshalk91tg This is not a config within values.yaml, for me the issue was resolved by changing the "Kubernetes Host" setting in the "Kubernetes Auth" engine in Vault. It was related to Vault in the end, not to VSO. Check my previous comment again.