search

openyurtio / openyurt

OpenYurt - Extending your native Kubernetes to edge(project under CNCF)
https://openyurt.io
Apache License 2.0
1.67k stars 386 forks source link

Openyurt using lot of bandwidth on edge nodes while communicating to cloud node #2038

Open nics90 opened 2 months ago

nics90 commented 2 months ago

What happened: We created 5 node cluster with 3 masters nodes running in cloud and 2 worker nodes running on Edge devices. Our Edge Devices uses 4G network to connect to internet, hence edge to cloud communication occur on that network. When we monitored the data usage on the 4G router we found that yurthub is consuming lot of data (100 MB per min), which means in a days it consuming approx 100 GB of data in 24 hours.

What you expected to happen: There is something wrong here since it should not consume that much of data. Is there any configuration to reduce yurthub data consumption.

How to reproduce it (as minimally and precisely as possible): Install OpenYurt Version 1.4.0 and Raven 0.4.0 and monitor the traffic from edge node to cloud node, it gets replicated.

Anything else we need to know?:

Environment:

others /kind question

rambohe-ch commented 2 months ago

@nics90 Thanks for raising the issue. Would you like to upload the metrics of yurthub component? and you can execute curl http://127.0.0.1:10267/metrics command on the edge node to get metrics of yurthub component.

The response traffic data is recorded in the metrics as request level.

nics90 commented 2 months ago

@rambohe-ch : Please find the attached files containing the metrics captured from two worker nodes. worker1.txt worker2.txt

rambohe-ch commented 2 months ago

@nics90 As the above metrics, there are too many requests that are sent by clients(such as longhorn-manager, go-http-client, calico-node, etc.)on worker nodes. and some requests are rejected by yurthub because of threshold is over.

rambohe-ch commented 2 months ago

@nics90 unfortunately, the traffic of these clients is not recorded in the metrics, you need to open cache for these agents in Yurthub.

nics90 commented 2 months ago

@rambohe-ch : Could you please help here and let me know how to open cache for these clients ? Thanks a lot!

rambohe-ch commented 2 months ago

@rambohe-ch : Could you please help here and let me know how to open cache for these clients ? Thanks a lot!

@nics90 you can configure cache_agents field as * in yurt-hub-cfg configmap in order to enable yurthub cache response for all agents.

https://github.com/openyurtio/openyurt/blob/37addb57d7a4d664832c6206cc72950c7151307a/charts/yurthub/templates/yurthub-cfg.yaml#L107

after you modify the above configuration, you need recreate all the pods on the node in order to construct cache from the beginning.

nics90 commented 1 month ago

@rambohe-ch : I tried the above config and it started caching all the clients, but unfortunately instead of decreasing the data usage it increased the usage. Could you please help here on priority basis.

Attached are the metrics. box2.txt box6.txt