search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.12k stars 807 forks source link

Can karmada-search list from member clusters in parallel when list request with Limit=0 ? #4798

Open sysucjl opened 1 month ago

sysucjl commented 1 month ago

What happened: When List pod with option Limit=0 by karmada-search, it will request all clusters in sequence. Therefore, the overall latency will increases linearly with the number of clusters: https://github.com/karmada-io/karmada/blob/c81649a423165bd660a3bf0ee3b1abd13b0f90e3/pkg/search/proxy/store/multi_cluster_cache.go#L263-L270

The following is the Trace log of a request：

image2024-4-2_12-19-6

We can see that there are two member clusters and ListFunc takes about 5 seconds per cluster, so "MultiClusterCache.List" takes 10 seconds in total.

What you expected to happen: I think that karmada-search can list from member clusters in parallel when list request with option Limit=0 to reduce the latency of serial processing.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

XiShanYongYe-Chang commented 3 weeks ago

@sysucjl thanks for your feedback!

Hi @ikaven1024, can you help take a look? /cc @ikaven1024

ikaven1024 commented 3 weeks ago

It can be improved in parallel. But in large cluster, limit=0 is not recommended. It will response a mass of objects, result in some problems such as timeout.

Couldu you share your scene？

ikaven1024 commented 3 weeks ago

similar to #4685

sysucjl commented 3 weeks ago

It can be improved in parallel. But in large cluster, limit=0 is not recommended. It will response a mass of objects, result in some problems such as timeout.

Couldu you share your scene？

I approve that limit=0 is not recommended in large cluster. We also rarely use limit=0 option in list request. But I think it is still worth to be improved in parallel , especially when karmada has many member clusters.