Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
When one of instance ES doesn't response by timeout then diamond doesn't send stats from all other instances too. It confused! Now when one of process has a problem it looks like as whole cluster are broken.
Do you have any logs that show where in the collector the timeout happened?
I am guessing that there is an uncaught exception that causes the entire collector to bomb out if a timeout happens
Hello! I have three instances of Elasticsearch on one host.
When one of instance ES doesn't response by timeout then diamond doesn't send stats from all other instances too. It confused! Now when one of process has a problem it looks like as whole cluster are broken.