Start watch {...} Failed to watch {...} loop when target cluster paused

Env Two eks clusters in seperate VPCs. eks1 with argocd eks2 with vClusters and apps deployed inside

Describe the bug I'm facing an issue with argocd controller which logs hundreds of Start watch {...} Failed to watch {...} It causes that NAT gateway active connections between VPCs are enormous, ~10-15x higher. The issue is visible each time vCluster is paused. As soon as argocd controller is restarted issue is gone. (no more Start watch... Failed to watch... logs and NAT active connections back to normal level)

To Reproduce Make target cluster unavailable (in my case vCluster paused) and observe controller logs. Then restart controller pod.

Expected behavior argocd controller doesn't need to be restarted when target cluster is unavailable

Version v2.11.2

Logs

level=info msg="Start watch WorkloadGroup.networking.istio.io on https://..." server="https://..."
level=info msg="Failed to watch AuthorizationPolicy.security.istio.io on https://...: Resyncing AuthorizationPolicy.security.istio.io on https://... due to timeout, retrying in 1s" server="https://..."
level=info msg="Start watch Predictor.serving.kserve.io on https://..." server="https://..."
level=info msg="Failed to watch ClusterIssuer.cert-manager.io on https://...: Resyncing ClusterIssuer.cert-manager.io on https://... due to timeout, retrying in 1s" server="https://..."
level=info msg="Failed to watch AllowedImages.constraints.gatekeeper.sh on https://...: Resyncing AllowedImages.constraints.gatekeeper.sh on https://... due to timeout, retrying in 1s" server="https://..."
level=info msg="Failed to watch ControllerRevision.apps on https://...: Resyncing ControllerRevision.apps on https://... due to timeout, retrying in 1s" server="https://..."

argoproj / argo-cd

Start watch {...} Failed to watch {...} loop when target cluster paused #19182