search

buildpacks-community / kpack

Kubernetes Native Container Build Service
Apache License 2.0
942 stars 161 forks source link

Different deployment mode to support good performance #992

Open showpune opened 2 years ago

showpune commented 2 years ago

Describe the problem/challenge you have I need to use kpack in a cluster to support build for 200 customers, and each customer has about 50 cnbimages. Totally we have more than 10000 cnbimages in the cluster.

  1. The reconcile queue is blocked because of so many cnbimages
  2. More pods can't help because each pod shares the same queue, they will conflict.
  3. If one customer start lots of cnbimages, other customers will wait, I hope each customer can has they own queue and quota.

Describe the solution you'd like Add deployment mode support for kpack, there are normal, cluster and namespace deployment mode support 1) Normal as current implementation 2) Cluster Model only reconcile the resource for cluster resource, like clusterstore , cluster stack and cluster builder 3) In Namespace deployment model, each deployment can define the work namespace environment parameter in deployment, and the deployment only reconcile the resource in the give namespace, like cnbimage and build In this way, the namespace deployment will process the CRDs in its own namespace, and cluster deployment process the cluster CRDs

Anything else you would like to add: There is similar issue for cartographer and kapp-controller, and there are some test data for kapp-controller as https://github.com/vmware-tanzu/carvel-kapp-controller/issues/792. Current situation image Namespace deployment after change image

tylerphelan commented 2 years ago

@showpune Thank you for this issue!

I'm curious, have you tried scaling the resource requests and limits for the kpack-controller deployment? Does it mitigate the problem at all?

showpune commented 2 years ago

Yes and I scaled both the request and limit of CPU to 2 and memory to 2048m, it helps little

tylerphelan commented 2 years ago

Thanks again for this issue. I think this is a relevant use case and problem. This would require a drastic change to the kpack installation and api which we will hopefully prioritize soon!

We also are happy to accept RFCs or PRs!

matthewmcnew commented 2 years ago

Totally we have more than 10000 cnbimages in the cluster.

@showpune The size of this deployment is unheard of with other kpack users. Could you provide a bit more context on what your environment looks like?

Are these all independent images? Are they deployed on the same cluster they are built? Would it be possible to run multiple clusters?

yilims commented 2 years ago

@matthewmcnew: Let me share more context for our environment. We have a cluster shared by 200 customers, the customers working on different projects/products and each customer produces around 50 CNB images, The images for different customers are totally independently and they use different tech stack. Customers are isolated by K8S namespace. For our business scenario, the customers have to share the same cluster.

@showpune and I are doing some investigation to scale up kpack controller pods to improve performance. The idea is:

  1. kpack controller pods are deployed in kpack namespace. the controller in kpack namespace reconciles cluster resources such as ClusterStore and ClusterStore.
  2. For every customer namespace, there's a kpack controller pod deployed to reconcile namespaced resources such as Builder and Image. The controller pod only reconciles the resources in the same namespace and it will NOT reconcile resources in the other namespaces.
  3. kpack webhook Service and Pod are only deployed in kpack namespace and it can validate the resources for all the namespaces. we have done a POC for this idea and will do some performance test to see the improvement. If everything works fine, we'd like to file a PR for this feature.

@matthewmcnew and @tylerphelan : Could you pls share your comment? Really appreciated for your help!

showpune commented 2 years ago

Hi @matthewmcnew ,

Now we want to isolate the resource for control plan and data plan into different K8S cluster and kpack will be installed in control plan K8S. 1) For data plan, every customer will own a K8S cluster 2) For control plan, we will focus on the work for build and deployment, to cutdown the cost, we want customers share a K8S cluster and customers have they own resource quota

pviraj59 commented 1 year ago

We are open to this idea. Would you be open to submit an RFC with proposed changes so that community can review them ?