search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.51k stars 490 forks source link

Multiple Carbon Cache Daemon #784

Closed rgleme closed 6 years ago

rgleme commented 6 years ago

Hello

I am trying to start 4 instances of carbon cache without sucess.

carbon.conf

[relay] USER = nginx LINE_RECEIVER_INTERFACE = 0.0.0.0 LINE_RECEIVER_PORT = 2003 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_PORT = 2004 RELAY_METHOD = consistent-hashing REPLICATION_FACTOR = 1 DESTINATIONS = 127.0.0.1:2014:1, 127.0.0.1:2024:2, 127.0.0.1:2034:3, 127.0.0.1:2044:4

[cache] USER = nginx GRAPHITE_URL = http://myurl.intranet LINE_RECEIVER_INTERFACE = 127.0.0.1 PICKLE_RECEIVER_INTERFACE = 127.0.0.1 CACHE_QUERY_INTERFACE = 127.0.0.1

[cache:1] LINE_RECEIVER_PORT = 2013 PICKLE_RECEIVER_PORT = 2014 CACHE_QUERY_PORT = 7012

[cache:2] LINE_RECEIVER_PORT = 2023 PICKLE_RECEIVER_PORT = 2024 CACHE_QUERY_PORT = 7022

[cache:3] LINE_RECEIVER_PORT = 2033 PICKLE_RECEIVER_PORT = 2034 CACHE_QUERY_PORT = 7032

[cache:4] LINE_RECEIVER_PORT = 2043 PICKLE_RECEIVER_PORT = 2044 CACHE_QUERY_PORT = 7042

I am getting the following error on webapp exception.log

Tue Jun 19 02:55:03 2018 :: Failed CarbonLink query 'prod.hongkong.hk-manager.a2-montesquieu7.tail.http_requests.counter.statuscode-201' Traceback (most recent call last): File "./graphite/readers.py", line 169, in fetch

File "./graphite/carbonlink.py", line 90, in query results = self.send_request(request) File "./graphite/carbonlink.py", line 118, in send_request conn = self.get_connection(host) File "./graphite/carbonlink.py", line 80, in get_connection connection.connect( (server, port) ) File "", line 1, in connect error: [Errno 111] Connection refused

I have tried to follow https://github.com/graphite-project/graphite-web/issues/1139

Still got the same error. Could you help me ? thanks

piotr1212 commented 6 years ago

Can you post your local_settings.py, especially the carbonlink_hosts setting

rgleme commented 6 years ago

Hello @piotr1212

CARBONLINK_HOSTS = ['127.0.0.1:7012', '127.0.0.1:7022','127.0.0.1:7032', '127.0.0.1:7042']

I also tried CARBONLINK_HOSTS = [] and with no parameter.

Other options in my local_settings.py are commented

Thank you for the reply

piotr1212 commented 6 years ago

Your instance ids are missing in the carbonlink_hosts, try:

CARBONLINK_HOSTS = ['127.0.0.1:7012:1', '127.0.0.1:7022:2','127.0.0.1:7032:3', '127.0.0.1:7042:4']
rgleme commented 6 years ago

@piotr1212

The same error appears, on webapp traceback

Tue Jun 19 11:04:05 2018 :: Failed CarbonLink query 'prod.hongkong.hk-manager.a2-montesquieu7.tail.http_requests.counter.statuscode-201' Traceback (most recent call last): File "./graphite/readers.py", line 172, in fetch cached_datapoints = CarbonLink.query(self.real_metric_path) File "./graphite/carbonlink.py", line 90, in query results = self.send_request(request) File "./graphite/carbonlink.py", line 118, in send_request conn = self.get_connection(host) File "./graphite/carbonlink.py", line 80, in get_connection connection.connect( (server, port) ) File "", line 1, in connect error: [Errno 111] Connection refused

ps axuf | grep carbon root 15897 0.0 0.0 103244 824 pts/2 S+ 11:06 0:00 _ grep carbon nginx 15866 2.7 0.0 226340 15804 ? S 11:06 0:00 /usr/bin/python /opt/graphite/bin/carbon-relay.py start nginx 15872 3.6 0.0 301316 14804 ? Sl 11:06 0:00 /usr/bin/python /opt/graphite/bin/carbon-cache.py --instance=1 start nginx 15879 3.5 0.0 300868 14692 ? Sl 11:06 0:00 /usr/bin/python /opt/graphite/bin/carbon-cache.py --instance=2 start nginx 15886 6.1 0.0 301240 14884 ? Sl 11:06 0:00 /usr/bin/python /opt/graphite/bin/carbon-cache.py --instance=3 start nginx 15893 6.5 0.0 301336 14856 ? Sl 11:06 0:00 /usr/bin/python /opt/graphite/bin/carbon-cache.py --instance=4 start

rgleme commented 6 years ago

@piotr1212 Now it worked :)

I restarted uwsgi process. Thank you for the help