search

deadtrickster / prometheus.erl

Prometheus.io client in Erlang
MIT License
341 stars 117 forks source link

speedup collection of counters and histograms #120

Closed RoadRunnr closed 3 years ago

RoadRunnr commented 3 years ago

Replace collection for counters and histograms with optimized version.

The collection functions for them had geometric complexity in the number of metrics. That made them unsusable for large datasets.

vkatsuba commented 3 years ago

Hi All,

Any updates about this improvement(merging/release etc)? Was retested fork paired with https://github.com/deadtrickster/prometheus.erl/pull/121 and with current PR from @RoadRunnr - tests shows that in case with large datasets the fetching of data occurs up to one second instead of 10 sec. as was before in version 4.8.0.

Regards, --V

essen commented 3 years ago

We've run into problems with large number of metrics so I'm asking @gerhard to take a look at this.

I am wondering how well Prometheus or Grafana performs with large number of metrics, how many metrics do you have and is everything OK otherwise?

vkatsuba commented 3 years ago

how many metrics do you have and is everything OK otherwise?

Peers Histogram ranges Buckets metrics per Scrape duration fork Scrape duration 4.8.0
7874 [10, 30, 50, 75, 100, 1000, 2000] 8 1.25s >10s