search

siimon / prom-client

Prometheus client for node.js
Apache License 2.0
3.15k stars 378 forks source link

custom metric state - could we keep it on disk? #562

Open RichardWright opened 1 year ago

RichardWright commented 1 year ago

We rely alot on metrics and the share of memory that prometheus uses is growing. Has it ever been suggested that counters/gauges could write to disk and retrieve that state for scraping?

Thanks for the great lib btw. We've recently moved off open telemetry and really appreciate the flexibility of the prom-client api, setting gauges is a great feature.

shappir commented 1 year ago

Saving to disk will make updating entires much more expensive - I don't think it's a viable approach in this case.

I would recommend looking at ways to reduce the cardinality of your data, especially histograms: reduce the number of labels, label value permutations, buckets.

There are ways in which the library itself could be improved, e.g. currently each histogram entry contains a dictionary of its label values, which is essentially a duplication of the data used as entry index key. A more memory efficient data structure could be used instead.

RichardWright commented 1 year ago

We gave up on histograms a while ago.

Memory efficieny would be appreciated. We have quite high cardinality as we run stateful, long lived, socket based applications and use the metrics as a way of shipping that state. Ideally we wouldn't, but an alternative pattern isn't obvious.