ec2-discovery not configured correctly at network.host

[tomerleib]

Current configuration provided in the user-data for the network.host is: network.host: ec2:privateIpv4,localhost

This cause the client node not to see the other nodes in the cluster with the following error: [o.e.d.z.ZenDiscovery ] [Elasticsearch-Master] not enough master nodes discovered during pinging.

Removing the localhost and adding ec2.endpoint partially resolves this issue, now the client node is able to ping all the masters however on Kibana it is reported that it cannot access elasticsearch on http://localhost:9200.

To solve this, on the client node I had to change it from ec2:privateIpv4 to 0.0.0.0 and everything started to work.

[synhershko]

Using 0.0.0.0 will make the node accept traffic on public IPs and that is not recommended

How exactly does removing localhost help? can you point to the relevant documentation?

[tomerleib]

I agree that it's not recommended. here's what I've found, the official docs for the ec2-discovery plugin states that: It’s important to define network.host as by default it’s bound to localhost. You can use core network host settings or ec2 specific host settings

Binding it to localhost result in only 127.0.0.1:9200 which result in hosts not able to ping each other. I was able to work only once the network.host on the client node is set to 0.0.0.0 and the other nodes in the cluster set to ec2

[synhershko]

This is why we bind to both ec2:privateIpv4,localhost. There might be something wrong in your configuration, a cluster deployed using this repo to AWS doesn't have this master error and has proper access from client nodes.

I will not change the bind host to 0.0.0.0.

[tomerleib]

I've found the real issue. The section of network.host is correct as you stated, the part that is needed is ec2.endpoint. It is set to use us-east-1 as default by the plugin, unless stated otherwise, so when the cluster is running on any other region, it will not work. Related to: https://github.com/elastic/elasticsearch/pull/27925 and also: https://github.com/elastic/elasticsearch/issues/27464

It should grab the correct endpoint if it's not set, however, due to open issue it does not. Sorry for the above :)

[synhershko]

That is incorrect. You should have set the variables correctly. Please follow the README. https://github.com/synhershko/elasticsearch-cloud-deploy/tree/master/terraform-aws#configurations

[synhershko]

Sorry, reading this again it seems like we are better adding the endpoint config to avoid error cases like you had, although we did successfully deploy on non-us-east-1 regions. I've opened https://github.com/synhershko/elasticsearch-cloud-deploy/issues/50