search

tkestack / kvass

Kvass is a Prometheus horizontal auto-scaling solution , which uses Sidecar to generate special config file only containes part of targets assigned from Coordinator for every Prometheus shard.
Apache License 2.0
614 stars 90 forks source link

feat: use stream model to reduce memory usage #81

Closed RayHuangCN closed 2 years ago

RayHuangCN commented 2 years ago

we use VictoriaMetrics lib to enable stream model in any scraping process. It reduce a log of memory useage of Coordinator and Sidecar.

In my benchmark case (50 targets, 1000w series), the memory usage of Coordinator < 100MB, and Sidecar with 240w max series only use about 50MB memory.

codecov[bot] commented 2 years ago

Codecov Report

Merging #81 (0623a55) into master (def3fe4) will increase coverage by 0.41%. The diff coverage is 88.14%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #81      +/-   ##
==========================================
+ Coverage   82.81%   83.23%   +0.41%     
==========================================
  Files          30       32       +2     
  Lines        1525     1569      +44     
==========================================
+ Hits         1263     1306      +43     
- Misses        160      161       +1     
  Partials      102      102
Impacted Files Coverage Δ
pkg/coordinator/coordinator.go 75.92% <ø> (ø)
pkg/coordinator/rebalance.go 78.00% <0.00%> (-0.54%) :arrow_down:
pkg/scrape/jobinfo.go 75.00% <75.00%> (ø)
pkg/scrape/reader.go 75.00% <75.00%> (ø)
pkg/sidecar/proxy.go 87.71% <83.33%> (+5.90%) :arrow_up:
pkg/scrape/scraper.go 93.22% <93.22%> (ø)
pkg/discovery/discovery.go 89.10% <100.00%> (+0.81%) :arrow_up:
pkg/explore/explore.go 87.50% <100.00%> (+0.59%) :arrow_up:
pkg/prom/config.go 82.35% <100.00%> (ø)
pkg/shard/kubernetes/shardmanager.go 66.66% <100.00%> (+3.25%) :arrow_up:
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 06a0b91...0623a55. Read the comment docs.