[BUG] CPU spiking for cattle-cluster-agent on downstream cluster and slowness in Rancher

[clp01]

Rancher Server Setup

• Rancher version: 2.7.2
• Installation option (Docker install/Helm Chart): Helm chart
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): v1.25.13+rke2r1
• Proxy/Cert Details:

Information about the Cluster

• Kubernetes version: v1.25.12+rke2r1
• Cluster Type (Local/Downstream): Downstream
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Custom

    Additional context cattle-cluster-agent-1.log cattle-cluster-agent-2.log

[clp01]

Any thoughts on the above?

In the hopes of seeing some improvement I have tried scaling the cattle-cluster-agent up to 3 replicas by running the following, but it is scaled back down again automatically (the additional pod goes straight into Terminating status):

kubectl scale --replicas=3 deployment cattle-cluster-agent -n cattle-system