search

PromPHP / prometheus_client_php

Prometheus instrumentation library for PHP applications
https://prometheus.io/docs/concepts/metric_types/
Apache License 2.0
423 stars 93 forks source link

Bad performance after updating to v2.6.1 #92

Open timmipetit opened 2 years ago

timmipetit commented 2 years ago

After updating to v2.6.1, which introduced https://github.com/PromPHP/prometheus_client_php/pull/91, we've seen a enormous drop in performance for our collection script. This is interesting, because the merge request is meant to increase performance.

Here's a screenshot from the average response time of our prometheus collection script:

Screenshot 2022-06-17 at 11 50 11
LKaemmerling commented 2 years ago

Hey @timmipetit,

can you try to downgrade to v2.6.0 just to make sure it is really related to the script? How many metrics do you expose? Might it be that the redis has a lot of keys?

timmipetit commented 2 years ago

Hi @LKaemmerling. We've downgraded back to v2.6.0, and performance is back to normal for us. We don't expose a lot of metrics, 6 histograms (with 16 buckets each) and 3 gauges (so no summaries). But our Redis instance is also used for sessions though, so we do have a lot of other keys (400k or so). The keys used for sessions have a different prefix. But since scan will iterate all keys, it will fetch all session keys (before discarding them again because they don't match)

grogy commented 2 years ago

I have to say the same from project https://supportbox.com/ - after upgrade to 2.6.1 so bad performance. Wipe storage (Redis) does not help.

LKaemmerling commented 2 years ago

Hello together,

i just released v2.6.2 with the revert of the given MR.

LKaemmerling commented 1 year ago

Hey @grogy & @timmipetit,

would it be possible that you test https://github.com/PromPHP/prometheus_client_php/pull/99/files against your workload (new StorageEngine RedisNG). This basically fixes the usage of KEYS and SCAN.

timmipetit commented 1 year ago

@LKaemmerling Thanks for the heads up. We'll test it

timmipetit commented 1 year ago

Tested on our workload, works fine!