search

external-secrets / external-secrets

External Secrets Operator reads information from a third-party service like AWS Secrets Manager and automatically injects the values as Kubernetes Secrets.
https://external-secrets.io/main
Apache License 2.0
4.45k stars 842 forks source link

ExternalSecret synchronization speed degradation #3258

Open alexandervidyaev opened 8 months ago

alexandervidyaev commented 8 months ago

As the number of ExternalSecrets objects increases, the synchronization speed of secrets with Vault significantly increases. Currently, there are about 600 ExternalSecrets in the cluster, and the synchronization time for an ExternalSecret can take up to 3 minutes. Is this normal behavior? How can performance be improved?

Skarlso commented 8 months ago

Can you try setting this value when you install external-secrets:

rootCmd.Flags().IntVar(&concurrent, "concurrent", 1, "The number of concurrent reconciles.")

See if that improves things.

Skarlso commented 8 months ago

In the chart:


# -- Specifies the number of concurrent ExternalSecret Reconciles external-secret executes at
# a time.
concurrent: 1
alexandervidyaev commented 8 months ago

Increasing this parameter had no effect. We also increased the number of replicas of all ESO components. This had no effect too.

Logs are also observed:

Waited for 10.19543532s due to client-side throttling, not priority and fairness, request: GET:https://172.18.0.1:443/api/v1/namespaces/

Can this be accompanied by slow performance of the API server?

pre commented 6 months ago

We are experiencing the exact same issue! We have ~800 namespaces with one ExternalSecret in each of them.

Logs have the Waited for NN.NNNN s due to client-side throttling, ... message while ExternalSecrets wait to be synced. Once the waiting period is over, ExternalSecrets become syncing again until the next throttle time hits.

We tried with both concurrent:1 (the default) and concurrent:10, but performance is very slow with both of them.

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Skarlso commented 3 months ago

What is your refresh interval?

pre commented 3 months ago

What is your refresh interval

It’s some time now, but IIRC we had 1 minute, 5 minutes and 30 minutes depending on the Secret.

pereiratps commented 1 month ago

Hi. Same here. Any update?

Skarlso commented 1 month ago

Not much really. In this case when there is client side throttling due to high rates of api requests. We could try and increase qps and burst but that would only be a temporary solution. There is also more aggressive caching options I don't know if you tried that yet?

There is a setting for it.

Try to play with these settings https://external-secrets.io/latest/api/controller-options/#core-controller-flags

See if any of them help?

Skarlso commented 1 month ago

These two especially


--enable-secrets-caching    boolean false   Enables the secrets caching for external-secrets pod.
--enable-configmaps-caching boolean false   Enables the ConfigMap caching for external-secrets pod.