Open blysik opened 6 years ago
The behavior of the offline host was odd. It was pinging, but ssh wouldn't connect. I wonder if it was still listening on the go-carbon/carbserver ports, but maybe it was a tcp timeout.
Could the dropped metrics and read timeouts be visualizing essentially tcp timeouts?
I am running is a slightly weird config while I try to migrate:
carbon-c-relay going to two separate clusters: a) 3 hosts in carbon_ch hash, go-carbon/carbonserver b) 4 hosts in jump fnva1 hash, go-carbon/carbonserver
carbonzipper is setup to read from all 7 host backends.
This morning, one of the jump_fnva1 machines went offline. At that time, I started to see really strange spikes of both write timeouts from carbon-c-relay, as well as carbonzipper reporting read timeouts.
See these graphs:
As soon as I go the go-carbon host back up, things went back to normal. But this behavior is strange, and not what I expected.
Does anyone know what could cause this?