mpelekh
commented
1 month ago The issue
In large clusters where Argo CD monitors numerous resources, the processing of watches becomes significantly slow—in our case (total k8s resources in cluster: ~400k, Pods: ~76k, ReplicaSets: ~52k), taking around 10 minutes. As a result, the Argo CD UI displays outdated information, impacting several features reliant on sync waves, like PruneLast. Eventually, the sheer volume of events from the cluster overwhelmed the system, causing Argo CD to stall completely.
To address this, we disabled the tracking of Pods and ReplicaSets, although this compromises one of the main benefits of the Argo CD UI. We also filtered out irrelevant events and tried to optimize various settings in the application controller. However, vertical scaling of the application controller had no effect, and horizontal scaling is not an option for a single cluster due to sharding limitations.
Issue causes
During the issue investigation, it was found that the problem lies in the following:
- resource lock contention
- slow performance of IterateHierarchy
Patched v2.10.9
v2.10.9 was patched with the following commits.
- https://github.com/argoproj/gitops-engine/commit/6b2984ebc47085852a7b63a0fd0b73c52e986217
- https://github.com/argoproj/argo-cd/commit/267f243a899483fea0a4e6a613c18f62bd342c7e
Though patches significantly improve performance, Argo CD still can not handle the load from large clusters.
In the screenshot, you can see one of the largest clusters. Here, the patched with the above commits v2.10.9 build is running. - till 12:50, pods and replica sets are disabled from tracking - from 12:50 to 13:34, pods and replica sets are enabled to be tracked - after 13:34, pods and replica sets are disabled from tracking As can be seen, once pods and rs are enabled to be tracked, the cluster event count falls close to zero, and reconciliation time increases drastically.   Number of pods in cluster: ~76k Number of rs in cluster: ~52k A more detailed comparison of different patched versions is added to this comment - https://github.com/argoproj/argo-cd/issues/8172#issuecomment-2278663389The potential reason is lock contention.
Here, a few more metrics were added, and it was found that when the number of events is significant, sometimes it takes ~5 minutes to acquire a lock, which leads to a delay in reconciliation. https://github.com/mpelekh/gitops-engine/commit/560ef00bcce9201083200f906f15bf1716fbfcc0#diff-9c9e197d543705f08c9b1bc2dc404a55506cfc2935a988e6007d248257aadb1aR1372 The suggested fix https://github.com/argoproj/gitops-engine/issues/602 to optimize the lock usage has not improved the issue in large clusters.
Avoid resources lock contention utilizing channel
Since we still have significant lock contention in massive clusters, and the approaches above didn’t resolve the issue, another approach has been considered. It is a part of this PR.
When we must acquire a write lock in each goroutine, we can’t handle more than one event at a time. What if we introduce the channel where all the received events are sent, and one goroutine is responsible for processing events in batch from the channel? In such a way, the locks from each goroutine are moved to the goroutine, which processes events from the channel. This means we would have only one place where the write lock is acquired; in such a way, we would get rid of the lock contention.
The fix results
 As can be seen from metrics, once the fixed version was deployed and Node, ReplicaSets, and Pods were enabled for tracking, the number of cluster events was stable and didn’t go down.Conclusions The fix shows significant performance improvements. We left Nodes, ReplicaSets, and Pods enabled on large clusters. ArgoCD UI is working smoothly. The original issue has been resolved - users can manage Pods and ReplicaSets on large clusters.
crenshaw-dev
sonarcloud[bot]
Problem statement is in https://github.com/argoproj/argo-cd/issues/8172#issuecomment-2277585238
The IterateHierrchyV2 significantly improved performance, getting us ~90% of the way there. But on huge clusters, we still have significant lock contention.
The fix in this pull request approaches the problem differently - it avoids lock contention by utilizing a channel to process events from the cluster.
More details are in the comments.