karmada apiserver memory upper limit affects federation cluster size

[xigang]

What do you think about this question?:

I have a question. Karmada will locally cache resource data defined by ResourceRegistry, such as pods, nodes, workloads, etc. If a federated cluster manages 100 member clusters, each with 5k nodes, will a single Karmada apiserver have a memory bottleneck?

How do we need to support larger federated clusters?

Environment:

• Karmada version:
• Kubernetes version:
• Others:

[xigang]

/cc @RainbowMango @XiShanYongYe-Chang @ikaven1024

[ikaven1024]

We have test it with 100 clusters, and 2 millions pods. FYI: https://github.com/karmada-io/karmada/issues/2518. If you have some other resources, having large quantity like pods, you can run searval karmada-search group, each group cache one or some kinds resources. And add a new component (may call it karmada-search-gateway), redirect client request to the target group due to the resource kind requested. Shows this architecture as below:

[XiShanYongYe-Chang]

you can run searval karmada-search group, each group cache one or some kinds resources.

That sounds great, but how to achieve that goal?

[ikaven1024]

May add some fields like group in RR. karmada-search-gateway and karmada-search-group watch these RRs. E.g.

apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: rr_pods
spec:
  group:
    pods: 
  targetCluster:{}
---
apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: rr_nodes_svcs
spec:
  group:
    nodes_svcs: 
  targetCluster:{}
---
apiVersion: search.karmada.io/v1alpha1
kind: ResourceRegistry
metadata:
  name: rr_foos
spec:
  group:
    foo: 
  targetCluster:{}

[XiShanYongYe-Chang]

Sounds like a good way to go.

[xigang]

@ikaven1024 But I understand that a single karmada-search group uses the memory of one machine, and there will still be memory bottlenecks, such as a very large number of pods resources.

[ikaven1024]

When search group reach its bottleneck, client informers also face the same problem. Our federation scale should have a upper bound, but we have not test it.

[xigang]

When search group reach its bottleneck, client informers also face the same problem. Our federation scale should have a upper bound, but we have not test it.

Thanks @ikaven1024 reply.