k8s-ci-robot
commented
1 year ago @lentzi90: The label(s) /label area/control-plane cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?
In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/8602):
>### What steps did you take and what happened?
>
>When scaling the number of workload clusters, the KCP controller stands out as [extremely CPU hungry](https://github.com/kubernetes-sigs/cluster-api/issues/8052#issuecomment-1456178857). I have been investigating this a bit closer now and found some hints about why this is. This is what I did:
>
>1. Set up a CAPI development environment with tilt and observability, profiling enabled
> ```yaml
> # tilt-settings.yaml
> default_registry: gcr.io/cluster-api-provider
> enable_providers:
> - docker
> - kubeadm-bootstrap
> - kubeadm-control-plane
> deploy_observability:
> - grafana
> - kube-state-metrics
> - prometheus
> debug:
> core:
> profiler_port: 40000
> kubeadm-control-plane:
> profiler_port: 40001
> kubeadm-bootstrap:
> profiler_port: 40002
> ```
>2. Observe metrics at idle with 0 clusters created and gather profiling data
>4. Create 10 clusters and let things settle
> 1. Note that I didn't apply any CNI in the workload clusters. This means they were re-queued more often than the sync interval because the nodes never became fully Ready, causing a bit higher load on the KCP controller. More clusters would anyway cause the same load though so I think this just helps the profiling.
>6. Compare metrics and gather profiling data again
>
>I took profiling samples using this command:
>
>```bash
>go tool pprof -http=:8080 -seconds=10 http://localhost:40001/debug/pprof/profile
>```
>
>I also created a simple [dashboard](https://github.com/kubernetes-sigs/cluster-api/files/11396432/dashboard.txt) in grafana to check CPU usage and workqueue metrics.
>
>Already at 10 clusters the KCP controller has a much higher CPU usage than the other controllers, as seen here:
>
>
>
>The flame graph shows quite clearly what it is about I think. Here is the [profile.pb.gz](https://github.com/kubernetes-sigs/cluster-api/files/11396512/profile.pb.gz), in case you want to investigate more.
>
>
>
>We generate new client certificates all the time. This is a quite CPU intensive operation. For large numbers of KCPs it becomes crazy!
>
>
>
>
>### What did you expect to happen?
>
>I'm hoping the KCP controller could be optimized to reduce the CPU requirements for larger numbers of clusters. Probably this will require some kind of cache for re-using client certificates used for connecting to the workload clusters. There is a TODO comment along these lines in the code that I found also: https://github.com/kubernetes-sigs/cluster-api/blob/7edbaf04263c77177ae68f4d64115d207fadb7eb/controlplane/kubeadm/internal/cluster.go#L104
>
>### Cluster API version
>
>Main branch (546f46464da13a98b87d469924b407fafe088df8)
>
>### Kubernetes version
>
>```console
>$ kubectl version
>WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
>Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
>Kustomize Version: v5.0.1
>Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-30T06:34:50Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
>```
>
>### Anything else you would like to add?
>
>Related issues:
>- #7308
>- #8052
>
>### Label(s) to be applied
>
>/kind bug
>/label area/control-plane
>
Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
killianmuldoon
sbueringer
fabriziopandini
lentzi90
What steps did you take and what happened?
When scaling the number of workload clusters, the KCP controller stands out as extremely CPU hungry. I have been investigating this a bit closer now and found some hints about why this is. This is what I did:
I took profiling samples using this command:
I also created a simple dashboard in grafana to check CPU usage and workqueue metrics.
Already at 10 clusters the KCP controller has a much higher CPU usage than the other controllers, as seen here:
The flame graph shows quite clearly what it is about I think. Here is the profile.pb.gz, in case you want to investigate more.
We generate new client certificates all the time. This is a quite CPU intensive operation. For large numbers of KCPs it becomes crazy!
What did you expect to happen?
I'm hoping the KCP controller could be optimized to reduce the CPU requirements for larger numbers of clusters. Probably this will require some kind of cache for re-using client certificates used for connecting to the workload clusters. There is a TODO comment along these lines in the code that I found also: https://github.com/kubernetes-sigs/cluster-api/blob/7edbaf04263c77177ae68f4d64115d207fadb7eb/controlplane/kubeadm/internal/cluster.go#L104
Cluster API version
Main branch (546f46464da13a98b87d469924b407fafe088df8)
Kubernetes version
Anything else you would like to add?
Related issues:
7308
8052
Label(s) to be applied
/kind bug /label area/control-plane