alibaba / open-local

cloud-native local storage management system for stateful workload, low-latency with simplicity
Apache License 2.0
467 stars 81 forks source link

在大规模场景下,降低agent对apiserver的访问压力 #253

Closed ypnuaa037 closed 7 months ago

ypnuaa037 commented 8 months ago

【背景问题】 在大规模集群场景下,发现agent对apiserver的压力很大。通过分析发现,主要源于两处:

  1. agent会不停的访问apiserver读取NodeLocalStorage,检测spdk配置,间隔只有100ms,因而对apiserver的访问频率 = 10 * N / s(N为节点数)
  2. agent会定时上报节点的NodeLocalStorage状态,无论NodeLocalStorage状态是否变化都会上报,上报周期是60s 其中,问题1是主要压力来源

【解决思路】

  1. 降低访问频率,不需要100ms这么频繁访问
  2. 只在NodeLocalStorage发生变化时才上报,因为大部分时间是没有变化的,不需要上报
CLAassistant commented 8 months ago

CLA assistant check
All committers have signed the CLA.

codecov-commenter commented 8 months ago

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (850073d) 32.35% compared to head (9275828) 31.94%.

Files Patch % Lines
pkg/agent/discovery/discovery.go 0.00% 22 Missing :warning:
pkg/csi/nodeserver.go 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #253 +/- ## ========================================== - Coverage 32.35% 31.94% -0.42% ========================================== Files 41 41 Lines 6426 6443 +17 ========================================== - Hits 2079 2058 -21 - Misses 4058 4096 +38 Partials 289 289 ``` | [Flag](https://app.codecov.io/gh/alibaba/open-local/pull/253/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/alibaba/open-local/pull/253/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba) | `31.94% <0.00%> (-0.42%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alibaba#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.