search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.5k stars 490 forks source link

Graphite-Carbon processes going down Abruptly with no error #854

Closed kondalarao15 closed 5 years ago

kondalarao15 commented 5 years ago

I've recently installed graphite and grafana on same instance and for some reason carbon processes going down with no error. I'm installed python 2.7.5 and graphite 1.1.5 versions.

I'm using the following carbon.conf file. carbon.txt

Current carbon process is up and running

image

But intermittently few of the carbon processes are going down and we're missing the metrics.

piotr1212 commented 5 years ago

There must be something in your carbon log or system logs or coredump, etc.

kondalarao15 commented 5 years ago

Under carbon-cache-b/query.log image. During that time carbin-cache-b process went down and I didn't see any exceptions in creates.log

piotr1212 commented 5 years ago

Nothing in your system logs, no core dumps? Only time I've seen carbon go down with nothing in it's log was due to the Linux OOM killer.

kondalarao15 commented 5 years ago

I don’t have any other system logs, if you see in the first image for carbon process(it’s showing a to h) process but abruptly few of the carbon processes are going down. Can we have a quick meeting regarding this issue.

piotr1212 commented 5 years ago

I don’t have any other system logs

What do you mean?

kondalarao15 commented 5 years ago

I've checked the system logs. These are the logs that are created on 05/12/2019 under /var/log folder. image

Under messages I've seen one error image And
image

piotr1212 commented 5 years ago

yup, oom-killer as expected. You are running out of memory, then Linux decides to kill a process.

DanCech commented 5 years ago

That will do it, you're trying to run 8 caches and 2 relays so you probably don't have enough memory for all that on your host.

kondalarao15 commented 5 years ago

Currently I’m using 8 core machine with 16gb ram. Could you please suggest me what’s the ideal cpu and RAM requirement for running 8carbon cache and 2 relays.

Or else with current 8core machine with 16gb RAM, how many carbon caches I can run to avoid carbon downtime. On Wed, May 15, 2019 at 3:46 PM Dan Cech notifications@github.com wrote:

That will do it, you're trying to run 8 caches and 2 relays so you probably don't have enough memory for all that on your host.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/graphite-project/carbon/issues/854?email_source=notifications&email_token=ADJ2UND5GXPETIR27N3SUNLPVRZCVA5CNFSM4HNDXSB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVP4WYA#issuecomment-492817248, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJ2UNCQIKKCU4AGUZTZBJDPVRZCVANCNFSM4HNDXSBQ .

kondalarao15 commented 5 years ago

Thanks piotr and DenCech. Currently I've carbon-cache-a to carbon-cache-h and two relays. Now I've commented carbon-cache-c to carbon-cache-h and restarted the graphite carbon. For some reason it's showing three carbon-cache-processes. image

Please see the following file for carbon.conf configuration file carbon.txt

piotr1212 commented 5 years ago

Could you please suggest me what’s the ideal cpu and RAM requirement for running 8carbon cache and 2 relays.

There are just too many variables to give an simple answer. Disk performance, number of metrics, metric interval, storage schema's, etc.