search

logsearch / logsearch-boshrelease

A BOSH-scalable Elasticsearch+Logstash+Kibana release
http://www.logsearch.io
Apache License 2.0
57 stars 46 forks source link

Ensure elasticsearch is receiving requests in ctl script before exiting #235

Closed bradylove closed 8 years ago

bradylove commented 8 years ago

Sometimes when doing a bosh deploy we have ran into an issue where the master node will deploy and report back healthy, then bosh will continue on to deploy the data nodes which will fail because the master is not accepting connections yet.

This change ensures that the elasticsearch node is accepting connections before continuing

cf-gitbot commented 8 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

cromega commented 8 years ago

@bradylove: interesting. Are you deploying with some funky settings perhaps? Which IAAS?

I have never seen this happening but I can definitely see the benefit of having such a check.

bradylove commented 8 years ago

@cromega We ran into this issue when deploying to AWS, It doesnt happen often but is annoying when it does.

Unfortunately we didn't save any logs from the data nodes when this happened.

bradylove commented 8 years ago

@cromega We are currently unable to reproduce the issue that was meant to fix. I am going to close the PR for now. We are going to explore having multiple master nodes which from the sound so it would resolve this problem and will solve some other issues for us.