search

python-diamond / Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
http://diamond.readthedocs.org/
MIT License
1.74k stars 602 forks source link

MongoDBCollector reduce sending data #750

Closed greggy closed 2 years ago

greggy commented 3 years ago

Hello!

I'm using diamond to collect mongodb metrics. I've build docker container with mongodb 3.6 and diamond. It sends metrics via handler GraphiteHandler and collector config is:

[[MongoDBCollector]]
enabled = True
hosts = localhost:27017
databases = production_db
ignore_collections = ^(.*_logs|sessions)
simple = True

Diamond flushes metrics every 10 seconds, in logs I see:

[2020-10-04 16:00:49,276] [ERROR] [MainThread:MongoDBCollector] Took too long to run! Killed!
...
[2020-10-04 16:03:48,101] [DEBUG] [MainThread:MongoDBCollector] Collection took 7172 ms

How to reduce number of sending metrics? Or I have another way to optimize it? Thank you!

shortdudey123 commented 2 years ago

I am guessing one of two things are happening. Either your mongo server is slow to respond to the collector queries or you have a lot of collections to iterate over. You should be able to see the slow queries on your mongo server. Reopen if this doesn't answer your question.