hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
471 stars 102 forks source link

Secret values starting with `.` result in empty response from Vault and no secret created #901

Closed roseDickinson closed 2 months ago

roseDickinson commented 2 months ago

Describe the bug We are using Vault with the Vault Secrets Operator in our Kubernetes cluster. I have been trying to track down for a few days now why some secrets in Vault were being added to the cluster fine but others were not. The only log that I found anywhere was in the events in the secret namespace/describe of the VaultStaticSecret's that were not being created Failed to read Vault secret: empty response from Vault, path="kvv2/data/prod/secret-name". The message itself wasn't very helpful so I just started random changes, it turned out to be the fact that in Vault the value of the secret started with a . as soon as I removed the . from the start of the secret it appeared in our Kubernetes cluster.

To Reproduce Steps to reproduce the behavior:

  1. Deploy Vault Secrets Operator to a cluster (version 0.8.1) using the default VaultConnection and default VaultAuth methods.
  2. Use the Kubernetes auth method.
  3. Setup Vault with kubernetes auth using the kvv2 secrets engine and using standard policy/role
  4. Add a secret with . at the start e.g. secret-name='.secret_value'
  5. Add a VaultStaticSecret pointing to the secret added in 4.
  6. Check the events and see the Failed to read Vault secret: empty response from Vault, path="kvv2/data/prod/secret-name error message
  7. Delete the . prefix from the secret value
  8. Check the cluster and see that the secret now exists

Application deployment:

We have Vault setup externally on a VM, no special config for it.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: vault-secrets-operator
  namespace: vault
spec:
  interval: 5m
  chart:
    spec:
      chart: vault-secrets-operator
      version: "0.8.1"
      sourceRef:
        kind: HelmRepository
        name: hashicorp
        namespace: flux-system
      interval: 1m
  values:
    defaultVaultConnection:
      enabled: true
      address: REDACTED
    defaultAuthMethod:
      enabled: true
      kubernetes:
        role: "vault-secrets-operator"
cat <<EOF | vault policy write vault-secrets-operator -
path "kvv2/data/*" {
  capabilities = ["read"]
}
EOF
vault write auth/kubernetes/role/vault-secrets-operator bound_service_account_namespaces='vault' bound_service_account_names='vault-secrets-operator-controller-manager,vault-agent-injector' policies='vault-secrets-operator' token_policies='vault-secrets-operator' ttl='9490h'
Name:         test4
Namespace:    vault
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultStaticSecret
Metadata:
  Creation Timestamp:  2024-08-30T13:35:36Z
  Finalizers:
    vaultstaticsecret.secrets.hashicorp.com/finalizer
  Generation:        3
  Resource Version:  192773357
Spec:
  Destination:
    Create:     true
    Name:       testing4
    Overwrite:  true
    Transformation:
  Hmac Secret Data:  true
  Mount:             kvv2
  Path:              secret-name
  Refresh After:     60s
  Type:              kv-v2
  Vault Auth Ref:    vault-auth
  Version:           2
Status:
  Last Generation:  3
  Secret MAC:      REDACTED
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  VaultClientError  106s (x3 over 3m19s)  VaultStaticSecret  Failed to read Vault secret: empty response from Vault, path="kvv2/data/secret-name"

Expected behavior The secret to be created in the cluster regardless of whether there is a . at the start. Or at the very least some documentation somewhere that explains restrictions in secret values (I haven't found this anywhere but apologies if it exists somewhere already).

Environment

roseDickinson commented 2 months ago

Nevermind - this was a misunderstanding of the purpose of the version field in the VaultStaticSecret spec. I was setting it to 2 because I thought it signified that it was the kv version 2 not the version of the secret to pull. Removing the . just happened to fix my issue (multiple times) because the "fixed" secrets were all the second versions of the secrets.