search

elastic / elasticsearch-cloud-aws

AWS Cloud Plugin for Elasticsearch
https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2
577 stars 181 forks source link

With a large number ec2 instances ec2 discovery times out #275

Open dat-vikash opened 8 years ago

dat-vikash commented 8 years ago

It seems with a large number of ec2 instances within a region, the ec2 discovery plugin will timeout

[2016-06-25 03:48:09,987][INFO ][discovery.ec2            ] [Zartra] Exception while retrieving instance list from AWS API: Unable to unmarshall response (ParseError at [row,col]:[174,26]
Message: Read timed out). Response Code: 200, Response Text: OK
[2016-06-25 03:49:21,390][INFO ][discovery.ec2            ] [Zartra] Exception while retrieving instance list from AWS API: Unable to unmarshall response (java.net.SocketTimeoutException: Read timed out). Response Code: 200, Response Text: OK

It would be great if we can configure this timeout either through environmental variables of system properties.

dadoonet commented 8 years ago

Thanks for reporting. Could you open this in Elasticsearch repo?

What is your version? How many Elasticsearch nodes you have? Do you use filtering to only get back Elasticsearch instances related to your project?

dat-vikash commented 8 years ago

I can open it in the elastic search repo, but I do think the issue lies with the aws ec2 discovery plugin. When I run the command:

aws ec2 describe-instances --region=us-east-1

on one of the ec2 instances within the cluster I do see a very large response time, on the order of 15-20 seconds. I'm guessing that with large node counts, the response time from this command goes up, so it would be great if we could increase the timeout via a setting.

ES Version : 2.3.3 I have 2 clusters : dev and qa each with 20 nodes. I launched dev on thursday evening with no problems and only seem to be having the problems with QA. Both are configured the same except for security groups.

I have not used filtering yet. Would filtering apply the the actual describe-instances command or would it only be used after the command has run? If it is applied to the describe-instances command, it could definitely be a workable solution.

Thanks!

dadoonet commented 8 years ago

This plugin don't live here anymore from 2.0.

dadoonet commented 8 years ago

IIRC filtering applies on AWS side so my guess is that you'll get back less instances. 20 nodes is a small number. Wondering why you have so bad response time.