cache filling up every 20 mins

grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter

Apache License 2.0

380 stars 107 forks source link

cache filling up every 20 mins #324

Closed pratikp09 closed 6 years ago

pratikp09 commented 6 years ago

I am running a setup of 2 carbon relay nodes which forward to graphite. I happen to notice recently the cache is filling up super fast every 15 mins. Restarting the nodes help clean it but i am unable to find the root cause as to why this is filling up so fast. Any input will be appreciated.

Civil commented 6 years ago

I'd start with checking that your destination nodes can handle the load. Check the CPU and Net on the storage nodes. Have a look if there are any obvious errors there (like metric drops, connection failures, etc)

How many points per second you are doing?

azhiltsov commented 6 years ago

carbon-c-relay produces the metrics about itself health. Health metrics usually starts with carbon.relay. Look for metricsReceived, metricsSent, metricsDropped there. Also destinations.$host:$port.metricsDropped could provide you some sort of visibility.