Open Larswa opened 7 months ago
Phew ... not sure if this is indeed a bug or not. Weekend came and gone, and unable to reproduce it. Now I just get a 403 solidly all the time wit the client jwt flow. With or without audience claims, and with or without the K8s api url as a separate audience claim as reported here https://github.com/hashicorp/vault-secrets-operator/issues/379
So why it actually worked and now its not ... I spend a day trying to reproduce this with no success. Moving to New Zealand to heard sheep very very soon.
Nope ... I got it working ... wrong url for kubernetes. Too long hours looking at the same problem. New Zealand is cancelled, and the issue still stands. I set it up as in the tutorial, I have to grant the default service account for app namespace the system:auth-delegator rights, and then its working. But only if there are no claims involved. The vault version running in docker is 1.15.4 and the VSO is 0.4.3
I get the same issue. Any news here?
I'm trying with an on-prem K8s 1.28 Vanilla and with EKS 1.28. Vault Version: 1.14.3
I get the same issue. Any news here?
I'm trying with an on-prem K8s 1.28 Vanilla and with EKS 1.28. Vault Version: 1.14.3
Hi Luana,
Can you get it to work w/o using an Audience claim?
I never got around to try and intercept tokens and check it the claim was actually in there or not ... but I didnt need to use that anyway so ...
What I couldn't read from the documentation was, that I needed to add this to enable the service accounts from the applications to get a token to talk to the external vault Kubernetes Authentication method. (in this case the pods run using the default service account, so if you use a specific one, you need to set that instead)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-auth-delegator
subjects:
- kind: ServiceAccount
name: default
namespace: kube-system
- kind: ServiceAccount
name: default
namespace: infrastructure
- kind: ServiceAccount
name: default
namespace: default
- kind: ServiceAccount
name: default
namespace: longhorn
- kind: ServiceAccount
name: default
namespace: rabbitmq
- kind: ServiceAccount
name: default
namespace: reflector
- kind: ServiceAccount
name: default
namespace: traefik
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
Yes, i removed the audience from the role in the auth method and from the vault CRD. This is my setup.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
annotations:
generation: 1
name: static-auth
namespace: app
spec:
kubernetes:
role: role1
serviceAccount: default
tokenExpirationSeconds: 600
method: kubernetes
mount: demo-auth-mount
status:
error: ""
valid: true
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
annotations:
name: vault-kv-app
namespace: app
spec:
destination:
create: true
name: secretkv
overwrite: false
hmacSecretData: true
mount: kvv2
path: webapp/config
refreshAfter: 30s
type: kv-v2
vaultAuthRef: static-auth
vault read auth/demo-auth-mount/config
Key Value
disable_iss_validation true
disable_local_ca_jwt false
issuer
kubernetes_ca_cert [REDACTED]
kubernetes_host [REDACTED]
pem_keys []
token_reviewer_jwt_set false
use_annotations_as_alias_metadata false
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
name: vault-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: default
namespace: app
So … trying out the Vault operator, and running through the demo in
GitHub - hashicorp-education/learn-vault-secrets-operator and the article here → The Vault Secrets Operator on Kubernetes | Vault | HashiCorp Developer
Except I am using an external Vault for these examples, and I grant the system:auth-delegator access to the service account described in the VaultAuth for my external vault Kubernetes auth. Like this
And
But this setup still gave me a 403 … untill I removed the audience from both my role definition in the Kubernetes Auth AND off course from my VaultAuth definition.
Without the audience there it just worked and whoopy… my secrets from vault were created.
Is there a known issue here with handling the audience claim when working against an a Vault external to Kubernetes?