search

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
469 stars 102 forks source link

permission denied error on VaultStaticSecret #729

Open ahsan-raza opened 6 months ago

ahsan-raza commented 6 months ago

Describe the bug I am implementing VaultStaticSecret with VSO. VaultDynamicSecret works fine with the same service account and permissions but VaultStaticSecret gives the following error:


URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
    * permission denied

    {"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"ns","name":"vault-static-secret-appname","uid":"0d8fe0f4-5172-4897-9995-314b5e52e040","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1758957"}, "reason": "VaultClientError"}
2024-05-07T18:33:41Z    DEBUG   events  Failed to read Vault secret: Error making API request.

To Reproduce Steps to reproduce the behavior:

  1. Deploy VSO with the following yaml file with the following VSO custom resources.

    defaultVaultConnection:
enabled: true
address: "https://vault.vault.svc.cluster.local:8200"
caCertSecret: "vault-ha-tls-vso"
tlsServerName: "tls-server-vault"
skipTLSVerify: true
    • secret file: 
      apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFME5UYzBOelUzTUI0WERUSTBNRFV3TVRFME5EVTFOMW9YRFRNME1EUXlPVEUwTkRVMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE5EVTNORGMxTnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLdnZmekFpbVRqMEU0ZFIrbm8vMytNVURyMVZSTERRWTdiRjZyaEsKelFUMEgyRGc1Y05TckxSSS9NUU5GTFc4czF4VUZPWlVTWGZzdEhvUjB2ditQZUdqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUS9PT05JU2d1c2Y3cUZGMGJPCjhOM3dYcDRBVVRBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBT1JRUmN2dEZEbFZYYkE2Qi9na2ZPM3BHcCtTOXkKeXVEaHovRmQ0b3Y2Z1FJZ2FVTHVqTE9wSnltOGg0ZjZWb29GQUVCeEx3TzZCWUl3UzkvWldKTTI4S2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
ca.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzNqYzZ1RkxXL0Uxd2wKWDRiL2dNVWUvVVloaitmQi9wa3Jib005Z0ZIZmYwL2h5aUQrRDE0RFNOZyt4Nk9ISWQwUGV4dGJQQkl2MWZXOQpHN3ZCUjNPY1M2cFhmbmtPS2ZsWEMzNVA2V28zNWxVWG1GVElhSTA3anlwcWhlY0JtR2NadXBKdXJRSm02Wk5kCktkREV4TFRCclNYSUZlVlJpUE9QV3lnWDgzK0ovRHZQV2hjOUV6YlR1Qys5d1pYYmRFNHNWbjZGQmd0YzJzZmUKTk1Wb3NkM3pSdXFOTnJGT3BVR0FVa3FRMUxhTTNmOWt4MmNRQlhSYWlDT0NmWUdKZFEycVo5bXlka2g2bDV2TQpkSkZ3QWNBRTNiSlJlc0NjMFdoZGtFYzluNkJkRllYU1ZnelhEeWl5MDd6Yno5c2w5cnJzelRrS0tHeXFlZzQxCklXVWpWdUF0QWdNQkFBRUNnZ0VBRENydW0wRnlDSWFzNzZqM2p1cnNtNkMxc1laZVdqdGtKY244NXdxMnJWNlMKdjAzblVOTFlkdFp4Ui96SWRVTmUvekVpNVFkTys2bWtQQTNCVjhTWDRFY2FJRDdZOXF4OHpxZ2dwQVZUWGxMRgphOFpwbDdlL093WkRJbmpFdkQ2V2dSRUtOczRwdXE3bVdhdnF0QXlIeTA5TTBmTjA5ajkzbm9FYXRUY1Y5VGNzCjFIWW5aZ09RMHNYUFoxNjlnUWdCdjZLQlBnalN3NnJIRFMzRXBSUzBhNVlXNnRpclRDaWFBbjhjL2l2NzdvSmgKclVKQzlXUWVHS3hySlB2cmVGUHpuekhCbmdOdFozandnbVZnbVl6cjBqKzhrWG5CbERKZmVaMmJqT0pGZ2tXNgprbHd1U0x3OWRlOVY0N2s3emt0Snh4SUl0SnNiYlJxVERyek1Oei81R1FLQmdRRHVoR1RMdDhXL1dWVmpmTndGCkNIV3N1aGhza0NyaUluNmZ3NnpURWVMYVZZeFV4UUw0NUFucGZvc3NRcitlc2Zmb2Z6cEx0dVowWTF0Qk5sTCsKRE9qYkQzMU0zT0FDeHNlSXBMY3lJSjRIVFl0YVFEQWMvdVpMdDBmT1M3OWpjRXlhWDdIUDZXNGFXTCtSdGdaQwpNZXlxcG1vSHZIZitYYzNuMnZxbFhmaVBKUUtCZ1FERkFoRkNRbWlpNlVxN0dHa3VEc1VObGs5Y3hwU1cwUWxvCkRrMytSOUlmVnFkSUdMV3ovMDdoeTVGNS9MbnU0eE56bjUvSWtBTGpUT2c4WUh0a2ZkQnpzTHNpclhjbmZtOVMKOEJTdWVoRExuZEZqemxBSVZ0bndoUGNjTU1rRndMQnFaT2xlblVxb29BYkE1N09UdFF2cm9aVDcxRnMvQ0gxcApiUm1KbzdoaWFRS0JnQmhEbWFscndZOHN2RllKbnF4WVp2Z3lXa2U1QnRQSVpJOUdMYSt2TXgrNDhqU3hjaEwxCng5aGNDalp2ZCtUaC8vRkQrQjg1dFFvRURZVjl6RVdSOUtKTVdoZldwR2RENTRxTUR5TG5WSXd6cEVpREhTSG0KQUhHVmJKV2MrUlJabVVGZkdNeW8xNDJRbDdSd0N3VHk1VHViQUZCWEtQSFVneHppRnZ0NXFzNk5Bb0dBWmtXcQpmQUZKbkxrTGFKRkZtUGxsNXFYUFVWUnRzdFdWMG9VS2pDVHd1Z0FjRzF0b2lLYlRabmh0Ynl4NXdiLzBTeVBrCnFSUEp6QUlTMWJVb21ZU1BBR0FRNWZHelY3ZFZSM01HNllUSXowdUFkaFdXaXAyN3loYmN5YSt4eStDNk5LRnEKWFFtK3hrYzgra3ZPZDdHWEhKM25YOHhnQ2hyNE1CREpIeUQxQ1hFQ2dZRUFzQ0g0K21lNU0yNmNzSFZITk5oUQpvZ1NCR2JXOGR4MU1XT09sbTlRb3hkdUdTcHpjcGR5Tk9HZU5HNG5Id0htRE5XeW9HYkVUUEhJT3lpb2xvc0g0CkNmRTU3dldxSW9RMnlXc3piKzZqdVZ5ZlJnQjhZd1ZqZ2dpL1R3bFlJSjR0NWZWbDdDTEV6TlBucnQzSzhJUGQKelhKMnRVMWNQOER4ekFuclAyNndXTW89Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: null
name: vault-ha-tls-vso
namespace: vault

    vault deployed with below values.yaml override:

    global:
enabled: true
tlsDisable: false
injector:
enabled: true
server:
extraEnvironmentVars:
  VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
  VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
  VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
volumes:
  - name: userconfig-vault-ha-tls
    secret:
     defaultMode: 420
     secretName: vault-ha-tls
volumeMounts:
  - mountPath: /vault/userconfig/vault-ha-tls
    name: userconfig-vault-ha-tls
    readOnly: true
standalone:
  enabled: false
affinity: ""
ha:
  enabled: true
  replicas: 3
  raft:
     enabled: true
     setNodeId: true
     config: |
        cluster_name = "vault-integrated-storage"
        ui = true
        listener "tcp" {
           tls_disable = 0
           address = "[::]:8200"
           cluster_address = "[::]:8201"
           tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
           tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
           tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
        }
        storage "raft" {
           path = "/vault/data"
        }
        disable_mlock = true
        service_registration "kubernetes" {}
  1. Any custom resources used for your secrets. VaultAuth.yml 
    apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: static-auth
namespace: ns
spec:
method: kubernetes
mount: kubernetes
kubernetes:
role: ns-role
serviceAccount: default
audiences:
  - vault
static-kv.yml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: ns
  name: vault-static-secret-appname
spec:
  vaultAuthRef: static-auth
  mount: kv-v2
  type: kv-v2
  path: ns/appname

  refreshAfter: 10s
  destination:
    create: true
    name: appname
  rolloutRestartTargets:
  - kind: Deployment
    name: ef-app
  1. Auth methods in Vault 
    
vault auth enable kubernetes

vault write auth/kubernetes/config \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ issuer="https://kubernetes.default.svc.cluster.local"


5. Roles, policies, in vault

vault secrets enable -version=2 kv-v2

vault policy write static-secret - << EOF path "kv-v2/data/ns/appname " { capabilities = ["read"] } EOF

vault write auth/kubernetes/role/ns-role \ bound_service_account_names=default \ bound_service_account_namespaces=ns \ policies=default,static-secret \ audience=vault \ ttl=24h


6. Error: (Retrieved from logs of vso pod)

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname Code: 403. Errors:

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ef-app
  name: ef-app
  namespace: ns
spec:
  selector:
    matchLabels:
      app: ef-app
  template:
    metadata:
      labels:
        app: ef-app
    spec:
      containers:
        - image: nginx
          name: nginx
          env:
          - name: SECRET_USERNAME
            valueFrom:
              secretKeyRef:
                name: appname
                key: username
          - name: SECRET_PASSWORD
            valueFrom:
              secretKeyRef:
                name: appname
                key: password
          - name: URI_TOKEN
            valueFrom:
              secretKeyRef:
                name: appname
                key: token
          resources: {}

kubectl describe VaultStaticSecret output.

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  VaultClientError  74s (x25 over 2m52s)  VaultStaticSecret  Failed to read Vault secret: Error making API request.

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Expected behavior VaultStaticSecret should create a kv secret in the namespace and inject in the pod.

Environment

Additional context

I followed this guide to deploy the vault https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls

Firstly, I assumed its the problem with tls so I deployed vault without tls and still the VSO was not able to create Static Secrets but was able to create dynamic secrets.

asttle commented 5 months ago

I have faced similar issue Reconciler error. Try to disable and re-enable kubernetes auth in vault server and configure and add the role and try again.

It worked for me.

ahsan-raza commented 5 months ago

@asttle Thanks for your comment and suggestion, I followed your steps, unfortunately, it did not work for me. I have deployed vault and 2 different server and also on minikube on my mac, I am facing the same issue

lusien88 commented 3 months ago

Hi,

We are currently facing the same problem. We've tried your solution @asttle, but some week later the problem returns and we have 4 environments facing the same problem.

No one from HashiCorp is assigned on this bug. Are there any update on this thread? Have you some information aboout it?

Thank you!

asttle commented 3 months ago

@ahsan-raza Configurations looks fine. Can you try to use a separate serviceAccount instead of using default to bind the vault and kubernetes.

@lusien88 Can you please share your configurations, so that i can have a look at it and help. Also refer to the video for setp by step implementation and check where it went wrong

https://www.youtube.com/watch?v=ECa8sAqE7M4&t=9s

ftrincal31 commented 3 months ago

Hi, same have @lusien88 with have same problem, when this bug are assignated to support Hashicorp team? Thank you !

111PYV111 commented 3 months ago

Hello, facing the same issue here. Exact same configuration as @ahsan-raza

Thanks for support !