permission denied error on VaultStaticSecret

Describe the bug I am implementing VaultStaticSecret with VSO. VaultDynamicSecret works fine with the same service account and permissions but VaultStaticSecret gives the following error:


URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
    * permission denied

    {"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"ns","name":"vault-static-secret-appname","uid":"0d8fe0f4-5172-4897-9995-314b5e52e040","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1758957"}, "reason": "VaultClientError"}
2024-05-07T18:33:41Z    DEBUG   events  Failed to read Vault secret: Error making API request.

To Reproduce Steps to reproduce the behavior:

Deploy VSO with the following yaml file with the following VSO custom resources.

defaultVaultConnection:
enabled: true
address: "https://vault.vault.svc.cluster.local:8200"
caCertSecret: "vault-ha-tls-vso"
tlsServerName: "tls-server-vault"
skipTLSVerify: true

secret file:

apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFME5UYzBOelUzTUI0WERUSTBNRFV3TVRFME5EVTFOMW9YRFRNME1EUXlPVEUwTkRVMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE5EVTNORGMxTnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLdnZmekFpbVRqMEU0ZFIrbm8vMytNVURyMVZSTERRWTdiRjZyaEsKelFUMEgyRGc1Y05TckxSSS9NUU5GTFc4czF4VUZPWlVTWGZzdEhvUjB2ditQZUdqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUS9PT05JU2d1c2Y3cUZGMGJPCjhOM3dYcDRBVVRBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBT1JRUmN2dEZEbFZYYkE2Qi9na2ZPM3BHcCtTOXkKeXVEaHovRmQ0b3Y2Z1FJZ2FVTHVqTE9wSnltOGg0ZjZWb29GQUVCeEx3TzZCWUl3UzkvWldKTTI4S2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
ca.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzNqYzZ1RkxXL0Uxd2wKWDRiL2dNVWUvVVloaitmQi9wa3Jib005Z0ZIZmYwL2h5aUQrRDE0RFNOZyt4Nk9ISWQwUGV4dGJQQkl2MWZXOQpHN3ZCUjNPY1M2cFhmbmtPS2ZsWEMzNVA2V28zNWxVWG1GVElhSTA3anlwcWhlY0JtR2NadXBKdXJRSm02Wk5kCktkREV4TFRCclNYSUZlVlJpUE9QV3lnWDgzK0ovRHZQV2hjOUV6YlR1Qys5d1pYYmRFNHNWbjZGQmd0YzJzZmUKTk1Wb3NkM3pSdXFOTnJGT3BVR0FVa3FRMUxhTTNmOWt4MmNRQlhSYWlDT0NmWUdKZFEycVo5bXlka2g2bDV2TQpkSkZ3QWNBRTNiSlJlc0NjMFdoZGtFYzluNkJkRllYU1ZnelhEeWl5MDd6Yno5c2w5cnJzelRrS0tHeXFlZzQxCklXVWpWdUF0QWdNQkFBRUNnZ0VBRENydW0wRnlDSWFzNzZqM2p1cnNtNkMxc1laZVdqdGtKY244NXdxMnJWNlMKdjAzblVOTFlkdFp4Ui96SWRVTmUvekVpNVFkTys2bWtQQTNCVjhTWDRFY2FJRDdZOXF4OHpxZ2dwQVZUWGxMRgphOFpwbDdlL093WkRJbmpFdkQ2V2dSRUtOczRwdXE3bVdhdnF0QXlIeTA5TTBmTjA5ajkzbm9FYXRUY1Y5VGNzCjFIWW5aZ09RMHNYUFoxNjlnUWdCdjZLQlBnalN3NnJIRFMzRXBSUzBhNVlXNnRpclRDaWFBbjhjL2l2NzdvSmgKclVKQzlXUWVHS3hySlB2cmVGUHpuekhCbmdOdFozandnbVZnbVl6cjBqKzhrWG5CbERKZmVaMmJqT0pGZ2tXNgprbHd1U0x3OWRlOVY0N2s3emt0Snh4SUl0SnNiYlJxVERyek1Oei81R1FLQmdRRHVoR1RMdDhXL1dWVmpmTndGCkNIV3N1aGhza0NyaUluNmZ3NnpURWVMYVZZeFV4UUw0NUFucGZvc3NRcitlc2Zmb2Z6cEx0dVowWTF0Qk5sTCsKRE9qYkQzMU0zT0FDeHNlSXBMY3lJSjRIVFl0YVFEQWMvdVpMdDBmT1M3OWpjRXlhWDdIUDZXNGFXTCtSdGdaQwpNZXlxcG1vSHZIZitYYzNuMnZxbFhmaVBKUUtCZ1FERkFoRkNRbWlpNlVxN0dHa3VEc1VObGs5Y3hwU1cwUWxvCkRrMytSOUlmVnFkSUdMV3ovMDdoeTVGNS9MbnU0eE56bjUvSWtBTGpUT2c4WUh0a2ZkQnpzTHNpclhjbmZtOVMKOEJTdWVoRExuZEZqemxBSVZ0bndoUGNjTU1rRndMQnFaT2xlblVxb29BYkE1N09UdFF2cm9aVDcxRnMvQ0gxcApiUm1KbzdoaWFRS0JnQmhEbWFscndZOHN2RllKbnF4WVp2Z3lXa2U1QnRQSVpJOUdMYSt2TXgrNDhqU3hjaEwxCng5aGNDalp2ZCtUaC8vRkQrQjg1dFFvRURZVjl6RVdSOUtKTVdoZldwR2RENTRxTUR5TG5WSXd6cEVpREhTSG0KQUhHVmJKV2MrUlJabVVGZkdNeW8xNDJRbDdSd0N3VHk1VHViQUZCWEtQSFVneHppRnZ0NXFzNk5Bb0dBWmtXcQpmQUZKbkxrTGFKRkZtUGxsNXFYUFVWUnRzdFdWMG9VS2pDVHd1Z0FjRzF0b2lLYlRabmh0Ynl4NXdiLzBTeVBrCnFSUEp6QUlTMWJVb21ZU1BBR0FRNWZHelY3ZFZSM01HNllUSXowdUFkaFdXaXAyN3loYmN5YSt4eStDNk5LRnEKWFFtK3hrYzgra3ZPZDdHWEhKM25YOHhnQ2hyNE1CREpIeUQxQ1hFQ2dZRUFzQ0g0K21lNU0yNmNzSFZITk5oUQpvZ1NCR2JXOGR4MU1XT09sbTlRb3hkdUdTcHpjcGR5Tk9HZU5HNG5Id0htRE5XeW9HYkVUUEhJT3lpb2xvc0g0CkNmRTU3dldxSW9RMnlXc3piKzZqdVZ5ZlJnQjhZd1ZqZ2dpL1R3bFlJSjR0NWZWbDdDTEV6TlBucnQzSzhJUGQKelhKMnRVMWNQOER4ekFuclAyNndXTW89Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: null
name: vault-ha-tls-vso
namespace: vault

vault deployed with below values.yaml override:

global:
enabled: true
tlsDisable: false
injector:
enabled: true
server:
extraEnvironmentVars:
  VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
  VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
  VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
volumes:
  - name: userconfig-vault-ha-tls
    secret:
     defaultMode: 420
     secretName: vault-ha-tls
volumeMounts:
  - mountPath: /vault/userconfig/vault-ha-tls
    name: userconfig-vault-ha-tls
    readOnly: true
standalone:
  enabled: false
affinity: ""
ha:
  enabled: true
  replicas: 3
  raft:
     enabled: true
     setNodeId: true
     config: |
        cluster_name = "vault-integrated-storage"
        ui = true
        listener "tcp" {
           tls_disable = 0
           address = "[::]:8200"
           cluster_address = "[::]:8201"
           tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
           tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
           tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
        }
        storage "raft" {
           path = "/vault/data"
        }
        disable_mlock = true
        service_registration "kubernetes" {}

cert used for vault:

apiVersion: v1
data:
vault.ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFME5UYzBOelUzTUI0WERUSTBNRFV3TVRFME5EVTFOMW9YRFRNME1EUXlPVEUwTkRVMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE5EVTNORGMxTnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLdnZmekFpbVRqMEU0ZFIrbm8vMytNVURyMVZSTERRWTdiRjZyaEsKelFUMEgyRGc1Y05TckxSSS9NUU5GTFc4czF4VUZPWlVTWGZzdEhvUjB2ditQZUdqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUS9PT05JU2d1c2Y3cUZGMGJPCjhOM3dYcDRBVVRBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBT1JRUmN2dEZEbFZYYkE2Qi9na2ZPM3BHcCtTOXkKeXVEaHovRmQ0b3Y2Z1FJZ2FVTHVqTE9wSnltOGg0ZjZWb29GQUVCeEx3TzZCWUl3UzkvWldKTTI4S2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
vault.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0akNDQW9pZ0F3SUJBZ0lRY2dKMmt6Wi96MWZuTE1ENitFSC9DREFLQmdncWhrak9QUVFEQWpBa01TSXcKSUFZRFZRUUREQmx5YTJVeUxYTmxjblpsY2kxallVQXhOekUwTlRjME56VTNNQjRYRFRJME1EVXdOakl3TXpVeApObG9YRFRJME1EZ3hOREl3TXpVeE5sb3dSekVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2WkdWek1TNHdMQVlEClZRUUREQ1Z6ZVhOMFpXMDZibTlrWlRvcUxuWmhkV3gwTG5OMll5NWpiSFZ6ZEdWeUxteHZZMkZzTUlJQklqQU4KQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdDQzT3JoUzF2eE5jSlYrRy80REZIdjFHSVkvbgp3ZjZaSzI2RFBZQlIzMzlQNGNvZy9nOWVBMGpZUHNlamh5SGREM3NiV3p3U0w5WDF2UnU3d1Vkem5FdXFWMzU1CkRpbjVWd3QrVCtscU4rWlZGNWhVeUdpTk80OHFhb1huQVpobkdicVNicTBDWnVtVFhTblF4TVMwd2EwbHlCWGwKVVlqemoxc29GL04vaWZ3N3oxb1hQUk0yMDdndnZjR1YyM1JPTEZaK2hRWUxYTnJIM2pURmFMSGQ4MGJxalRheApUcVZCZ0ZKS2tOUzJqTjMvWk1kbkVBVjBXb2dqZ24yQmlYVU5xbWZac25aSWVwZWJ6SFNSY0FIQUJOMnlVWHJBCm5ORm9YWkJIUForZ1hSV0YwbFlNMXc4b3N0TzgyOC9iSmZhNjdNMDVDaWhzcW5vT05TRmxJMWJnTFFJREFRQUIKbzRHdE1JR3FNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFNQmdOVgpIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRkQ4NDQwaEtDNngvdW9VWFJzN3czZkJlbmdCUk1GUUdBMVVkCkVRUk5NRXVDRUNvdWRtRjFiSFF0YVc1MFpYSnVZV3lDS0NvdWRtRjFiSFF0YVc1MFpYSnVZV3d1ZG1GMWJIUXUKYzNaakxtTnNkWE4wWlhJdWJHOWpZV3lDQnlvdWRtRjFiSFNIQkg4QUFBRXdDZ1lJS29aSXpqMEVBd0lEU0FBdwpSUUlnQWlqdDBSOHlSSmlDTGFIckNqWitLSUJIeEtpNVNoOGIwWmZkVzQxVVRPb0NJUUNXc2xZbjdUdllvaEh2CmJWQjAxb2hxZ1lHeTBaeXBGSUdiT3FYbXl5eW1Odz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
vault.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzNqYzZ1RkxXL0Uxd2wKWDRiL2dNVWUvVVloaitmQi9wa3Jib005Z0ZIZmYwL2h5aUQrRDE0RFNOZyt4Nk9ISWQwUGV4dGJQQkl2MWZXOQpHN3ZCUjNPY1M2cFhmbmtPS2ZsWEMzNVA2V28zNWxVWG1GVElhSTA3anlwcWhlY0JtR2NadXBKdXJRSm02Wk5kCktkREV4TFRCclNYSUZlVlJpUE9QV3lnWDgzK0ovRHZQV2hjOUV6YlR1Qys5d1pYYmRFNHNWbjZGQmd0YzJzZmUKTk1Wb3NkM3pSdXFOTnJGT3BVR0FVa3FRMUxhTTNmOWt4MmNRQlhSYWlDT0NmWUdKZFEycVo5bXlka2g2bDV2TQpkSkZ3QWNBRTNiSlJlc0NjMFdoZGtFYzluNkJkRllYU1ZnelhEeWl5MDd6Yno5c2w5cnJzelRrS0tHeXFlZzQxCklXVWpWdUF0QWdNQkFBRUNnZ0VBRENydW0wRnlDSWFzNzZqM2p1cnNtNkMxc1laZVdqdGtKY244NXdxMnJWNlMKdjAzblVOTFlkdFp4Ui96SWRVTmUvekVpNVFkTys2bWtQQTNCVjhTWDRFY2FJRDdZOXF4OHpxZ2dwQVZUWGxMRgphOFpwbDdlL093WkRJbmpFdkQ2V2dSRUtOczRwdXE3bVdhdnF0QXlIeTA5TTBmTjA5ajkzbm9FYXRUY1Y5VGNzCjFIWW5aZ09RMHNYUFoxNjlnUWdCdjZLQlBnalN3NnJIRFMzRXBSUzBhNVlXNnRpclRDaWFBbjhjL2l2NzdvSmgKclVKQzlXUWVHS3hySlB2cmVGUHpuekhCbmdOdFozandnbVZnbVl6cjBqKzhrWG5CbERKZmVaMmJqT0pGZ2tXNgprbHd1U0x3OWRlOVY0N2s3emt0Snh4SUl0SnNiYlJxVERyek1Oei81R1FLQmdRRHVoR1RMdDhXL1dWVmpmTndGCkNIV3N1aGhza0NyaUluNmZ3NnpURWVMYVZZeFV4UUw0NUFucGZvc3NRcitlc2Zmb2Z6cEx0dVowWTF0Qk5sTCsKRE9qYkQzMU0zT0FDeHNlSXBMY3lJSjRIVFl0YVFEQWMvdVpMdDBmT1M3OWpjRXlhWDdIUDZXNGFXTCtSdGdaQwpNZXlxcG1vSHZIZitYYzNuMnZxbFhmaVBKUUtCZ1FERkFoRkNRbWlpNlVxN0dHa3VEc1VObGs5Y3hwU1cwUWxvCkRrMytSOUlmVnFkSUdMV3ovMDdoeTVGNS9MbnU0eE56bjUvSWtBTGpUT2c4WUh0a2ZkQnpzTHNpclhjbmZtOVMKOEJTdWVoRExuZEZqemxBSVZ0bndoUGNjTU1rRndMQnFaT2xlblVxb29BYkE1N09UdFF2cm9aVDcxRnMvQ0gxcApiUm1KbzdoaWFRS0JnQmhEbWFscndZOHN2RllKbnF4WVp2Z3lXa2U1QnRQSVpJOUdMYSt2TXgrNDhqU3hjaEwxCng5aGNDalp2ZCtUaC8vRkQrQjg1dFFvRURZVjl6RVdSOUtKTVdoZldwR2RENTRxTUR5TG5WSXd6cEVpREhTSG0KQUhHVmJKV2MrUlJabVVGZkdNeW8xNDJRbDdSd0N3VHk1VHViQUZCWEtQSFVneHppRnZ0NXFzNk5Bb0dBWmtXcQpmQUZKbkxrTGFKRkZtUGxsNXFYUFVWUnRzdFdWMG9VS2pDVHd1Z0FjRzF0b2lLYlRabmh0Ynl4NXdiLzBTeVBrCnFSUEp6QUlTMWJVb21ZU1BBR0FRNWZHelY3ZFZSM01HNllUSXowdUFkaFdXaXAyN3loYmN5YSt4eStDNk5LRnEKWFFtK3hrYzgra3ZPZDdHWEhKM25YOHhnQ2hyNE1CREpIeUQxQ1hFQ2dZRUFzQ0g0K21lNU0yNmNzSFZITk5oUQpvZ1NCR2JXOGR4MU1XT09sbTlRb3hkdUdTcHpjcGR5Tk9HZU5HNG5Id0htRE5XeW9HYkVUUEhJT3lpb2xvc0g0CkNmRTU3dldxSW9RMnlXc3piKzZqdVZ5ZlJnQjhZd1ZqZ2dpL1R3bFlJSjR0NWZWbDdDTEV6TlBucnQzSzhJUGQKelhKMnRVMWNQOER4ekFuclAyNndXTW89Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: null
name: vault-ha-tls
namespace: vault

Any custom resources used for your secrets. VaultAuth.yml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: static-auth
namespace: ns
spec:
method: kubernetes
mount: kubernetes
kubernetes:
role: ns-role
serviceAccount: default
audiences:
  - vault

static-kv.yml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: ns
  name: vault-static-secret-appname
spec:
  vaultAuthRef: static-auth
  mount: kv-v2
  type: kv-v2
  path: ns/appname

  refreshAfter: 10s
  destination:
    create: true
    name: appname
  rolloutRestartTargets:
  - kind: Deployment
    name: ef-app

Auth methods in Vault
```
vault auth enable kubernetes
```

vault write auth/kubernetes/config \ token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ issuer="https://kubernetes.default.svc.cluster.local"


5. Roles, policies, in vault

vault secrets enable -version=2 kv-v2

vault policy write static-secret - << EOF path "kv-v2/data/ns/appname " { capabilities = ["read"] } EOF

vault write auth/kubernetes/role/ns-role \ bound_service_account_names=default \ bound_service_account_namespaces=ns \ policies=default,static-secret \ audience=vault \ ttl=24h


6. Error: (Retrieved from logs of vso pod)

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname Code: 403. Errors:

1 error occurred:
- permission denied
{"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"ns","name":"vault-static-secret-appname","uid":"0d8fe0f4-5172-4897-9995-314b5e52e040","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1758957"}, "reason": "VaultClientError"} 2024-05-07T18:33:41Z DEBUG events Failed to read Vault secret: Error making API request.

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ef-app
  name: ef-app
  namespace: ns
spec:
  selector:
    matchLabels:
      app: ef-app
  template:
    metadata:
      labels:
        app: ef-app
    spec:
      containers:
        - image: nginx
          name: nginx
          env:
          - name: SECRET_USERNAME
            valueFrom:
              secretKeyRef:
                name: appname
                key: username
          - name: SECRET_PASSWORD
            valueFrom:
              secretKeyRef:
                name: appname
                key: password
          - name: URI_TOKEN
            valueFrom:
              secretKeyRef:
                name: appname
                key: token
          resources: {}

kubectl describe VaultStaticSecret output.

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  VaultClientError  74s (x25 over 2m52s)  VaultStaticSecret  Failed to read Vault secret: Error making API request.

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Expected behavior VaultStaticSecret should create a kv secret in the namespace and inject in the pod.

Environment

Ubuntu server 22-04
Kubernetes version:
- v1.28.9+rke2r1
vault-secrets-operator version: 0.6.0
vault version: 1.16.1

Additional context

I followed this guide to deploy the vault https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls

Firstly, I assumed its the problem with tls so I deployed vault without tls and still the VSO was not able to create Static Secrets but was able to create dynamic secrets.

hashicorp / vault-secrets-operator

permission denied error on VaultStaticSecret #729