search

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
471 stars 102 forks source link

VaultStaticSecrets not resuming secret syncing post a Vault seal event #840

Open MysticalMount opened 4 months ago

MysticalMount commented 4 months ago

Describe the bug VaultSaticSecret doesnt resume syncing post a Vault being sealed, and then unsealed. Vault instance is external to the cluster and was successfully unsealed.

Expected VaultSaticSecret to resume syncing post unseal.

To Reproduce

  1. Set up a VaultConnection and VaultAuth in the operator namespace.
  2. Create a VaultStaticSecrert using default connection and auth
  3. Confirm secret syncing
  4. Terminate a Vault instance
  5. Bring back up a Vault instance
  6. Unseal
  7. VaultStaticSecret will report 503 Vault is sealed permanenty

Application deployment:

    defaultVaultConnection:
      enabled: true
      address: "http://xxx:8200"
    defaultAuthMethod:
      enabled: true
      namespace: ""
      method: kubernetes
      mount: xxx
      kubernetes:
        role: "xxx"
        serviceAccount: vault-auth

Vault operator was restarted. No errors in the controller/operator logs pre or post the restart. Problem persisted for all VaultStaticSecrets.

Expected behavior Expected VaultStaticSecret to resume post Vault becoming re-available, it seemed to stop after roughly 10 minutes but this is a best guess.

Environment

Additional context Add any other context about the problem here.

MysticalMount commented 4 months ago

I think whats happening here is the secret is being refreshed, but no event is generated if the secret already exists. Deletion of the target secret resource, post Vault's status becoming available again (i.e. successful connection and unsealed) - does seem to happen automatically.

However Ive only tested this post a connection error, whereupon I realised this, versus a 503/Vault is sealed but likely the behaviour is the same.

alexthaii commented 2 months ago

I've also hit this bug during a seal/unseal event.

Restarting VSO didn't help. Deleting either the target Secret or the VaultStaticSecret will make the VaultStaticSecret start syncing successfully again.

Environment: Image: hashicorp/vault-secrets-operator:0.8.1 Kubernetes: 1.27.13