Configure single namespace for application-controller

[Bondza]

Summary

An application-controller seem to use all namespaces to list and watch applications today. It would be useful to be able to configure a single namespace to list and watch applications from.

Motivation

We run argocd in one cluster that then manages more than 15k applications across 670 clusters. We run one application-controller per cluster, i.e. one shard per cluster. With 15k applications in one namespace listing applications is quite slow and takes > 30s. This is quite a big cause of problems for us. For example when rolling application-controllers during an update they will be rate limited by the Kubernetes apiserver, if they are all restarted at the same time. Also it means that it will take a while before a new application-controller comes up since it needs to list and watch all applications before the controller becomes ready.

We tried moving applications into namespaces, and create one namespace per cluster. Listing of applications for one cluster would then be fast, even though listing all of them would be as slow as before. We thought this might work since the application-controller takes --namespace as an option, this is then used when setting up the Application informer. But this does not seem to work without also setting --application-namespaces which forces application-controllers to list and watch applications from all namespaces. Also setting the --namespace option also means that it is necessary to move some ConfigMaps and Secrets to the namespace to make the application-controller run.

Proposal

Make it possible to list and watch applications from a single namespace. One option might be to check the length of the --application-namespaces option, if the length is one and --namespace is not set then watchNamespace could be set to the only entry in the --application-namespaces option.

Or if this will not work, maybe a new option (maybe --application-namespace) could be added to the application-controller and set as config on the application-controller. And then be used to configure the watchNamespace if none of the other namespace related options are configured.

[andrii-korotkov-verkada]

Which version of ArgoCD are you using? Try upgrading to v2.13, as there was IterateHierarchyV2, which greatly sped up building of the application tree.

I'm not sure sharding by namespace is possible easily, since application can change a namespace via manifest and then which controller shard should process it?