search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.5k stars 490 forks source link

Getting CarbonClientFactory related issue #861

Closed caldavid closed 4 years ago

caldavid commented 5 years ago

Hi,

I've been experiencing the data delay, and found committed-points has been always below Metrics-received, then found below error in the client.log:

Destination is up: my_IP:2014:None CarbonClientProtocol(my_IP:2014:None)::connectionLost Connection to the other side was lost in a non-clean fashion. CarbonClientFactory(1my_IP:2014:None)::clientConnectionLost (my_IP:2014) Connection to the other side was lost in a non-clean fashion. Destination is down: my_IP:2014:None (1/5) CarbonClientFactory(my_IP:2014:None)::startedConnecting (my_IP:2014) CarbonClientProtocol(my_IP:2014:None)::connectionMade CarbonClientFactory(my_IP:2014:None)::connectionMade (CarbonClientProtocol(my_IP:2014:None))

wondering what went wrong?

Thanks, David

caldavid commented 5 years ago

Screen Shot 2019-07-29 at 12 03 31 PM here is what the trend looks like

caldavid commented 5 years ago

some more log from console.log:

Stopping factory CarbonClientFactory(my_IP:2014:None) Starting factory CarbonClientFactory(my_IP:2014:None) <twisted.internet.tcp.Connector instance at 0x7f41e6fea2d8> will retry in 2 seconds

piotr1212 commented 5 years ago

overloaded caches? This is your relay log right?

caldavid commented 5 years ago

Thanks for looking into the issue!

This is from both graphite/storage/log/carbon/console.log, and graphite/storage/log/carbon/client.log

Does graphite support multiple carbon-cache instances running in parallel on the same box?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.