VaultStaticSecret: support instant event-driven updates

[tvoran]

Support for triggering a VaultStaticSecret (VSS) sync from Vault events instead of just on a refresh interval. Adds a new parameter in a new config block (SyncConfig.InstantUpdates) that enables VSS Reconcile() to spawns a goroutine with a websocket client that receives event notifications from Vault, and triggers Reconcile() for the VSS using the controller's source channel. Uses a registry based on go-cache to store which VSS' have event watchers running along with the event watcher's context cancel function and other control metadata.

Requirements:

• Vault Enterprise 1.16+
• Event policy permissions (see example below)

Caveats:

• There may be issues with the eventing client going stale with non-periodic Vault tokens (to be addressed in a future update)

Documentation update: https://github.com/hashicorp/vault/pull/27668

The websocket event client is built with the VaultStaticSecret's auth, so extra permissions will be needed on the policy for the associated VaultAuth role.

Example for a kv-v2 secret "campaign" with the engine mounted under "kv-marketing" in the "us-west-org" namespace:

$ VAULT_NAMESPACE=us-west-org vault policy write marketing-read-only -<<EOF
path "kv-marketing/data/campaign" {
   capabilities = ["read", "list", "subscribe"]
   subscribe_event_types = ["kv*"]
}

path "sys/events/subscribe/kv*" {
   capabilities = ["read"]
}

EOF

And the corresponding VaultStaticSecret:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: static-demo
  namespace: nicecorp
spec:
  namespace: us-west-org
  vaultAuthRef: event-demo-auth
  mount: kv-marketing
  type: kv-v2
  path: campaign
  refreshAfter: 1h   # <-- long refresh interval as a fallback
  destination:
    name: campaign-secret
    create: true
  syncConfig:
    instantUpdates: true  # <-- new option

Vault event subscription status and errors are emitted as k8s events on the VaultStaticSecret object. For example:

$ kubectl describe vaultstaticsecret static-demo -n nicecorp
...
Events:
  Type     Reason               Age                   From               Message
  ----     ------               ----                  ----               -------
  Normal   SecretSynced         2m37s                 VaultStaticSecret  Secret synced
  Normal   SecretRotated        2m37s                 VaultStaticSecret  Secret synced
  Warning  EventWatcherError    16s (x18 over 2m37s)  VaultStaticSecret  Error while watching events: 
failed to connect to vault websocket: error returned when opening event stream web socket to 
wss://vault.vault.svc.cluster.local:8200/v1/sys/events/subscribe/kv%2A?json=true, ensure 
VaultAuth role has correct permissions and Vault is Enterprise version 1.16 or above: {"errors":["permission denied"]}
  Normal   EventWatcherStarted  6s                    VaultStaticSecret  Started watching events

[Floppy012]

Is there any chance to implement this for Static Credentials of DB Secret Engines too? Or is this too far out of scope for this PR? (https://github.com/hashicorp/vault-secrets-operator/issues/494#issuecomment-1840689503)

[tvoran]

Hi @Floppy012, DB Engine events are out of scope for this PR, but they're on our list for the future.