search

sensu-plugins / sensu-plugins-graphite

This plugin provides native Graphite instrumentation for monitoring, including: replication status, various Graphite data queries, mutators, and handlers
http://sensu-plugins.io
MIT License
22 stars 45 forks source link

Missing metrics #48

Closed mendicm closed 7 years ago

mendicm commented 7 years ago

Hello, I don't know if this is the good place for this question because I don't have at this moment any clue of this issue... After I increased the number of connected clients sending metrics I'm getting this kind of graphs:

captura de pantalla 2017-03-17 a las 18 26 25

There is something throttling the data flow but I don't know where. My configuration is: sensu-redis-graphite-grafana. Redis is used for the transport too. The cpu and ram on the server is ok, disk write speed is normal (up to 3MB/s). Not found errors on sensu clients/server logs... Any help would be appreciated. Thank you!

mendicm commented 7 years ago

FYI: graphite is running on docker container and the metrics are collected every 10s

mendicm commented 7 years ago

Same problem on graphite-web:

captura de pantalla 2017-03-17 a las 19 50 16
GhostLyrics commented 7 years ago

I am seeing the same thing even though I don't use this plugin. I'm using a plain TCP handler instead. I initially thought it might be due to the sample interval being too low, but you said you're sampling every 10 seconds?

mendicm commented 7 years ago

I'm using a tcp handler too but I posted the issue here because I don't know where to post it... And yes I'm sampling every 10s and downsampling throught graphite after 7 days of retention.

mendicm commented 7 years ago

Finally I found one of the client machines with the sensu-server service running, seems that the problem was this.