Kibana reported that ES is not working in EC2

[neubineshkuttan]

Hi, I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all.

I also get multiple errors on x-pack services with the same errors on license information.

What else is needed to check in order to get it working?

[synhershko]

Please provide the configuration you used to deploy this

[AAber]

Hi, Got the same issue, the problem is that the discovery-ec2 plugin is not working.

As a temp fix you can remove the ec2 discovery: network.host: _ec2:privateIpv4_,localhost plugin.mandatory: discovery-ec2 cloud.node.auto_attributes: true cluster.routing.allocation.awareness.attributes: aws_availability_zone discovery: zen.hosts_provider: ec2 ec2.groups: sg-XXXXXXXXXXXXXXXX ec2.host_type: private_ip ec2.tag.Cluster: default-staging-es-cluster ec2.availability_zones: eu-west-1a,eu-west-1b,eu-west-1c ec2.protocol: http # no need in HTTPS for internal AWS calls

And setup simple discovery: network.host: _site_,localhost discovery.zen.ping.unicast.hosts: ["ip-172-30-2-22.eu-west-1.compute.internal", "ip-172-30-0-139.eu-west-1.compute.internal", "ip-172-30-1-64.eu-west-1.compute.internal", "ip-172-30-1-66.eu-west-1.compute.internal", "ip-172-30-0-217.eu-west-1.compute.internal"]

Let me know what configuration files are needed to debug the issue / ec2 discovery, and I will post.

[neubineshkuttan]

Thanks for the info. I will check this. Any permanent solution for this problem?

[neubineshkuttan]

Please provide the configuration you used to deploy this. network.host: site,localhost discovery.zen.ping.unicast.hosts: ["ip-172-30-2-22.eu-west-1.compute.internal", "ip-172-30-0-139.eu-west-1.compute.internal", "ip-172-30-1-64.eu-west-1.compute.internal", "ip-172-30-1-66.eu-west-1.compute.internal", "ip-172-30-0-217.eu-west-1.compute.internal"]

@synhershko - The issue got resolved with the above change. Is this an expected manual change?

please refer the config details.

bootstrap.memory_lock: true node.name: ${HOSTNAME}

action.destructive_requires_name: true indices.fielddata.cache.size: 1% # default is unbounded cluster.name: ELKCluster

discovery.zen.minimum_master_nodes: 2

only data nodes should have ingest and http capabilities

node.master: true node.data: false node.ingest: false http.enabled: false xpack.security.enabled: false xpack.monitoring.enabled: true path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch

network.host: ec2:privateIpv4,localhost plugin.mandatory: discovery-ec2 cloud.node.auto_attributes: true cluster.routing.allocation.awareness.attributes: aws_availability_zone discovery: zen.hosts_provider: ec2 ec2.groups: sg-xxxxxxxxxxxxxxxxx ec2.host_type: private_ip ec2.tag.Cluster: default-ELKCluster ec2.availability_zones: us-east-1a,us-east-1b,us-east-1c,us-east-1e ec2.protocol: http # no need in HTTPS for internal AWS calls

[kgosietsileM]

Hi, I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all.

I also get multiple errors on x-pack services with the same errors on license information.

What else is needed to check in order to get it working?

Hi, @neubineshkuttan

I also experienced the same issue, I found that the discovery plugin defaults to ec2.us-east-1.amazonaws.com as the EC2 endpoint if the EC2 endpoint for discovery is not explicitly set.

https://github.com/elastic/elasticsearch/issues/27464

My ES Cluster was up and running after I explicitly set the region for my es cluster in my elasticseach.yml configuration. Please see the example below.

    zen.hosts_provider: ec2
    ec2.groups: ${security_groups}
    ec2.endpoint: ec2.eu-west-1.amazonaws.com
    ec2.host_type: private_ip
    ec2.tag.Cluster: ${es_environment}
    ec2.availability_zones: ${availability_zones}
    ec2.protocol: http # no need in HTTPS for internal AWS calls```

[AAber]

Super cool!! I updated my config and now it works like a charm.

Thx.

[synhershko]

I'm thinking we should add this to user_data.sh:

ec2.endpoint: ec2.${aws_region}.amazonaws.com

where aws_region gets the value from var.aws_region

thoughts?

[synhershko]

Also, this issue is already open as https://github.com/synhershko/elasticsearch-cloud-deploy/issues/50

[neubineshkuttan]

@synhershko , If ec2.endpoint: ec2.${aws_region}.amazonaws.com is included as of user_data.sh then the "commented on Sep 21" will be resolved?

[neubineshkuttan]

Hi, I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all. I also get multiple errors on x-pack services with the same errors on license information. What else is needed to check in order to get it working?

Hi, @neubineshkuttan

I also experienced the same issue, I found that the discovery plugin defaults to ec2.us-east-1.amazonaws.com as the EC2 endpoint if the EC2 endpoint for discovery is not explicitly set.

elastic/elasticsearch#27464

My ES Cluster was up and running after I explicitly set the region for my es cluster in my elasticseach.yml configuration. Please see the example below.

    zen.hosts_provider: ec2
    ec2.groups: ${security_groups}
    ec2.endpoint: ec2.eu-west-1.amazonaws.com
    ec2.host_type: private_ip
    ec2.tag.Cluster: ${es_environment}
    ec2.availability_zones: ${availability_zones}
    ec2.protocol: http # no need in HTTPS for internal AWS calls```
``

Hi @synhershko , @kgosietsileM , @AAber

After adding the endpoint, the issue still exist. please refer the below configs.

network.host: ec2:privateIpv4,localhost plugin.mandatory: discovery-ec2 cloud.node.auto_attributes: true cluster.routing.allocation.awareness.attributes: aws_availability_zone discovery: zen.hosts_provider: ec2 ec2.groups: sg-XXXXXXXXXXXXXXXXX ec2.endpoint: ec2.us-west-2.amazonaws.com ec2.host_type: private_ip ec2.tag.Cluster: xxxxxxxxxxxxxxxxxxxxxxx ec2.availability_zones: us-west-2a,us-west-2b,us-west-2c ec2.protocol: http # no need in HTTPS for internal AWS calls

Few other observations, 1) Elasticsearch service still fails. 2) Error in log file as below ---

[ERROR][o.e.b.Bootstrap ] Exception java.lang.IllegalStateException: missing mandatory plugins [discovery-ec2], found plugins [] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:177) ~[elasticsearch-6.4.2.jar:6.4.2]

[yibinlin]

Maybe related: in packer this step seemed failing in master branch:

amazon-ebs: Provisioning with shell script: install-cloud-plugin.sh amazon-ebs: grep: /var/lib/dhcp/dhclient.eth0.leases: No such file or directory

[BenAHammond]

This worked for @neubineshkuttan and I. Thank you for the fix.

[tsimonyan]

It seems issue was in cloud plugin not being installed. Should be fixed in master version.