Closed abelhoula closed 1 month ago
@maximilianbraun @MisterMX Any recommendations on addressing this memory leak issue?
this was due to the huge nb of secrets in our clusters (secrets created by Trivy operator), so after the cleanup memory was back to normal.
@abelhoula the large number of secrets was in the cluster where the provider was running or argocd?
@maximilianbraun both the provider and argocd, more than 50k secrets. I think the same goes for crossplane as well, after the removal of trivy & cleanup of secrets everything is back to normal. https://github.com/crossplane/crossplane/issues/5272
What happened?
We have identified a performance bottleneck in the provider ArgoCD, particularly in a large scale environment. The system is experiencing a memory leak due to a large number of Kubernetes Secret objects. Profiling indicates that methods such as
k8s.io/api/core/v1.(*Secret).Unmarshal
and caching operations likek8s.io/client-go/tools/cache.(*Reflector).list.func1
have the highest memory usage. Notably, even with just one project resource, the memory usage spiked to 1 GB.How can we reproduce it?
This issue can be reproduced in an environment with a large number of Kubernetes Secret objects. The high memory usage is observed when unmarshaling these secrets and during cache reflection operations.
What environment did it happen in?
Crossplane version: 1.14.6 Crossplane Provider argocd version: 0.7.0
![image](https://github.com/crossplane-contrib/provider-argocd/assets/45313572/a107c488-9270-4028-a3fc-546530746874)
Root Cause
The high memory usage is likely due to unmarshaling large amounts of Kubernetes Secret objects being pulled from an API server, which suggests that there may be numerous Secrets or that the Secrets themselves are large. Additionally, cache reflection operations appear to be involved, indicating that the client is actively watching or listing numerous resources, leading to increased memory use, potentially due to storing the objects in memory.
Recommended Fix
Please consider the following optimizations options: