[Q] Why one of the 8 go-carbon nodes in cluster is experience too much read load on CPU while others are normal?

[nadeem1701]

We have a carbon-graphite cluster with 2 carbon-c-relays and 8 go-carbon nodes. Recently, we are noticing alarms for high CPU load on one of the worker nodes. Upon investigation. Upon investigation, we found that go-carbon is putting too much I/O read load. It is having read load approximately equivalent to the other 7 together.

It is to be noted that we do not use go-carbon to fetch the metrics from the cluster. We use graphite-app (python version) for this purpose. It is not causing the IO issue as we have done per-process CPU analysis.

I need help identifying RCA for this abnormality in that one of the worker nodes with the same HW and SW configuration behaves differently.

go-carbon version: 0.14.0 graphite-webapp: 1.2.0

[deniszh]

Hi @nadeem1701

Different load means that read or write load is skewed somehow, and usually that happens because of read and write configuration (i.e. your relay and graphite-web) and not go-carbon itself. Are you sure that node 7 is participating in reads coming from graphite web? Could you please share (anonimized) config for both your relay and graphite-web?

[deniszh]

Ah, I misread graph. Node 7 almost getting no traffic and node 2 is overloaded. Well, default graphite sharding is not really uniform, it's better to use jump hash for that. But please note that graphite-web do not support jump hash directly, you'll need to connect graphite-web to carbonservers (poprt 8080) on go-carbon using CLUSTER_SERVERS then.

[nadeem1701]

Thank you @deniszh for your very quick response.

The metric values in the legend are the last values at a given time, so we cannot say that Node#7 is getting the least/no traffic. gets a relatively fair amount of traffic (cyan-colored line).

We do not use carbonserver to fetch metrics from the cluster. We have graphite-webapp running on all worker nodes and graphite-webapp with relay configurations on relay-nodes. We can way that we use go-carbon to write metrics and graphite-webapp to read them. If Python based webapp was causing read load on CPU, it could have been understandable. In this case, go -carbon is stressing CPU with READ. We use fnv1a for hashing and did not expect this much imbalance.

relay-configs: #################################################### cluster carbon fnv1a_ch dynamic 172.22.1.1:2003=a 172.22.1.2:2003=b 172.22.1.3:2003=c 172.22.1.4:2003=d 172.22.1.5:2003=e 172.22.1.6:2003=f 172.22.1.7:2003=g 172.22.1.8:2003=h ; match * send to carbon ; statistics submit every 60 seconds reset counters after interval ; #################################################

Graphite-web ################################################# LOG_ROTATION = True LOG_ROTATION_COUNT = 1 DEFAULT_XFILES_FACTOR = 0 CLUSTER_SERVERS = ["172.22.1.1", "172.22.1.2", "172.22.1.3", "172.22.1.4", "172.22.1.5", "172.22.1.6", "172.22.1.7", "172.22.1.8"] USE_WORKER_POOL = True REMOTE_STORE_MERGE_RESULTS = True CARBONLINK_HASHING_TYPE = 'fnv1a_ch' FUNCTION_PLUGINS = [] #################################################

[deniszh]

Hi @nadeem1701 Thanks! Could you please share your go-carbon config please? Tbh I'm bit confused how your setup works. What process listens port 80 on 172.22.1.x servers?

[nadeem1701]

We have graphite-web running on 80 port of all 172.22.1.x (worker nodes). That is where graphite-web of relay server connects to fetch metrics. This might put some context:

and go-carbon configs:

[common] user = "carbon" graph-prefix = "carbon.agents.{host}" metric-endpoint = "local" metric-interval = "1m0s" max-cpu = 3

[whisper] data-dir = schemas-file = aggregation-file = workers = 6 max-updates-per-second = 0 max-creates-per-second = 0 hard-max-creates-per-second = false sparse-create = false flock = false enabled = true hash-filenames = true

[cache] max-size = 1000000 write-strategy = "max"

[udp] enabled = false

[tcp] listen = "0.0.0.0:2003" enabled = true buffer-size = 0

[pickle] enabled = false

[carbonlink] listen = "0.0.0.0:7002" enabled = true read-timeout = "30s"

[grpc] enabled = false

[tags] enabled = false

[carbonserver] enabled = false

[pprof] enabled = false

[deniszh]

@nadeem1701 : ah, got it. Is main graphite-web has same config as you post above?

[nadeem1701]

Yes. the graphite-web configs shared earlier are relay-graphite-web's. It queries the graphite-web running on all 8 worker nodes and returns collected metrics.

[deniszh]

if local graphite-web share same set of IPs I think you need to set REMOTE_EXCLUDE_LOCAL=True to avoid loops IIRC. And main graphie-web can be excluded then - you can set requests to all graphite-webs to balance load. But besides that I see no issues with you config TBH and I don't know why it can cause disbalance.