kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.93k stars 1.46k forks source link

Do you have an indication how much resources the controller require and should plan and how it can scale ? #3258

Closed yousefshaban00 closed 1 year ago

yousefshaban00 commented 1 year ago

what is recommended and should be taken into consideration when identifying and specifying resources for a controller?

and for that, it will need some memory and CPU. I assume it will also watch pods and namespace, in additional to doing a reconciliation of the current state i.e. check if the load balancers have been changed outside of Kubernetes, we should get an understanding of how many resources the controller requires.

resources: {} We usually recommend not specifying default resources and to leave this as a conscious choice for the user. This also increases chances charts run on environments with little resources, such as Minikube. If you do want to specify resources, uncomment the following lines, adjust them as necessary, and remove the curly braces after 'resources:'. limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi

oliviassss commented 1 year ago

@yousefshaban00, we haven't monitored the resource usage yet during the performance test of the LB controller, but I'd say the memory and CPU limit would really depend on your use case. I did a quick test on my setup where there are 2 aws-load-balancer-controller pods, and during the scaling it can surge to:

Would you be able to install the k8s metrics server and monitor the resource usage with your controller scaling? Let me know if it helps with your case

yousefshaban00 commented 1 year ago

@oliviassss, Thank you very much. However, we have two use cases:

First use case: We need to provision 100 to 500 namespaces using one AWS ALB controller. We define an ingress group name annotation for each namespace.

Second use case: We need to provision 10 to 50 AWS ALBs through one AWS ALB controller. We define an LB for each namespace.

Question: How much CPU and memory resources should be configured? It is crucial for us to define the exact values considering the cost implications.

M00nF1sh commented 1 year ago

@yousefshaban00 The memory only grows with the size & count of your Kubernetes Objects spec. We have done optimizations to minimize the memory usage(e.g. we cache minimal pod information in memory).

I'd suggest you first run the controller without cpu/memory limit, and monitor it via prometheus and set the limit to a proper value based on the size you observed in your cluster.

oliviassss commented 1 year ago

@yousefshaban00 Hi, will the above info help with your use case? I'm closing this issue as of now, feel free to reach out or reopen if you have more questions. Thanks