Vault Secrets Operator memory usage way too high

[cascadia-sati]

Describe the bug We have Vault 1.16.1 and Vault Secrets Manager 0.7.1 running in our dev, sandbox, and prod EKS 1.26 clusters. The vault-secrets-manager pod is using a lot more memory than the default 128 MB limit defined in the chart's value file, despite only syncing around 60 secrets. At startup this sometimes even spikes to well above 256 MB, so we had to set the limit to 512 MB.

Recently the VSO pod in the dev cluster started getting OOMKilled again, so we had to increase the limit once more. For some reason the VSO pod in dev is using more than double the memory of the ones running in sandbox and prod, even though the number of secrets is relatively the same across all envs.

Dev:

$ k top pod vault-secrets-operator-controller-manager-5cf49f6444-f6j7w
NAME                                                         CPU(cores)   MEMORY(bytes)
vault-secrets-operator-controller-manager-5cf49f6444-f6j7w   6m           426Mi

$ k get vaultstaticsecret -A | wc -l
      61

Sandbox:

$ k top pod vault-secrets-operator-controller-manager-66d4c79c6d-wfb8j
NAME                                                         CPU(cores)   MEMORY(bytes)
vault-secrets-operator-controller-manager-66d4c79c6d-wfb8j   6m           189Mi

$ k get vaultstaticsecret -A | wc -l
      63

Prod:

$ k top pod vault-secrets-operator-controller-manager-57cccf65c5-mm4ql
NAME                                                         CPU(cores)   MEMORY(bytes)
vault-secrets-operator-controller-manager-57cccf65c5-mm4ql   8m           157Mi

$ k get vaultstaticsecret -A | wc -l
      60

Here is dev at its peak:

$ kubectl top pod vault-secrets-operator-controller-manager-5cf49f6444-97skb
NAME                                                         CPU(cores)   MEMORY(bytes)
vault-secrets-operator-controller-manager-5cf49f6444-97skb   3m           839Mi

I tried upgrading VSO in dev to 0.9.0, but nothing changed.

Our setup is very basic out of the box.

Each secret has no more than a couple dozen values, most of them much less.

My questions are:

1) Why does the VSO pod use more memory than the default limit in such a basic scenario?

2) Why does the VSO pod need so much memory in general?

3) Why does the dev VSO pod need more than twice the memory of the pod in the other envs despite syncing the same number of secrets? Or how can we find this out?

[tvoran]

Hi @cascadia-sati, that level of memory usage does seem a bit high. Are VaultStaticSecret's the only secret type in use? What kind of auth methods are being used? Are there any errors showing up in the VSO logs?

Since you mentioned the dev VSO pod is using twice the memory as other envs, are there differences in secret types, auth methods, error rates, etc. between the envs?