yabeda-rb / yabeda-prometheus

Adapter to expose metrics collected by Yabeda plugins to Prometheus
MIT License
113 stars 18 forks source link

Speed of /metrics depends on how many file are in /tmp/prometheus_direct_file_store #31

Closed silva96 closed 4 days ago

silva96 commented 1 week ago

I configured metrics using the DataStore at /tmp/prometheus_direct_file_store at first it was responding super fast, then after few months, my metrics stopped being accessible in grafana and after debugging I realizing the metrics endpoint was responding in 14 seconds, but default prometheus timeout is 10 seconds.

I then deleted all the files inside /tmp/prometheus_direct_file_store and then the metrics endpoint started to respond super fast again.

is there a way to prevent this?

Or do I have to Manually delete the files each week or so?

Envek commented 4 days ago

Prometheus Ruby client README has some recommendations, in particular:

If you have this issue, one way to do this is to run code similar to this as part of you initialization:

Dir["#{app_path}/tmp/prometheus/*.bin"].each do |file_path|
 File.unlink(file_path)
end

Unfortunately, nothing can be done at yabeda-prometheus level, I believe (but if you have any ideas please share!)

silva96 commented 3 days ago

Thanks for pointing that documentation out. I implemented it