search

ViaQ / watches-cli

:watch: CLI tool to pull statistics from Elasticsearch
Apache License 2.0
1 stars 1 forks source link

Is node sniffing working? #22

Open lukas-vlcek opened 7 years ago

lukas-vlcek commented 7 years ago

I am not sure if -s, --sniff option works as expected or if we setup it correctly. See sniffing doc for explanation first and then see the source code.

I did basic perf test with three node ES cluster. The watches tool was configured to collect nodes_stats data like this for about 20 minutes:

watches nodes_stats \
  -d $TTL -i 3 \
  -sblt \ # <- see the use of '-s' here
  --url=${ESURL} \
  > watches-nodes-stats-stdout.txt

It is important to note that the ES cluster was initially started as a single node cluster and other two nodes were added during the time (within the first ~3 minutes or so).

After pbench processed the output file and generated charts it can be seen that only a single node (out of three nodes) was dealing with external http traffic. See screenshot:

screen shot 2017-01-23 at 16 13 56

There should be two sources of external HTTP traffic:

1) A couple of parallel stream2es processes indexing wikipedia snapshot. The truth is that all stream2es processes were configured to connect to ES using the same HTTP IP address (TODO: investigate if stream2es can do it's own sniffing). So this source of external traffic is expected to put all the load to a single node.

2) A couple of watches processes, each polling different stats from ES cluster. It is truth that all were started using the same --url=${ESURL} value but at the same time all of them were using -s option. So I would expect that at least some external HTTP traffic should be shared by other ES nodes. But this is not the case.

richm commented 7 years ago

Are you using the openshift aggregated logging Elasticsearch? If so, it has to go through some sort of route/service which "hides" (hopefully load balancing) the actual ES nodes.

lukas-vlcek commented 7 years ago

No openshift. Just three ES nodes on my localhost started using <ES_HOME>/bin/elasticsearch -d command forming a single cluster. I am able to query each node separately using different PORT# like:

I can also query each node from other machine because I bind ES process to the machine internal public IP address. I need to look at this... (I will try to upload my scripts that I use to setup the cluster and index wikipedia data to it).