Elasticsearch - No Living Connections

[nexflo]

This has been mentioned before here and there..but im certain this is still a bug or connections arent pooled correctly?

Im doing the following: (it indexes 6.000.000 docs...one by one..(dont wanna use bulk imports..so lets ignore this topic for a second) I'm even waiting for an index to be complete to call the next one...still at around 16.000 entries...i always get the no more living connections error.

I tried with keepalive: true/false etc. but it seems its a connection pooling issue no? iprange = array of item esearch = Elasticsearch client.

function doItem(i){
    process.stdout.write("Importing item:  " + i + "\r");
    var doc = {};
    doc.index = "geoip";
    doc.type = "location";
    doc.body = iprange[i];
    esearch.index(doc, function (err, resp) {
        if(err)
            trace.info(err);

        i++;

        if(i < len) {
            doItem(i);
        }
    });
}

doItem(1);

[spalger]

I just published a version of the client that includes agent keep alive if you're interested: https://github.com/elastic/elasticsearch-js/issues/476

[toiye]

I tried a variety of these solutions, but it was just turning off sniffing that ended up working for me. I am using Elastic cloud service, not AWS'.

[mbifulco]

@toiye What did you to do turn off sniffing?

[toiye]

I previously had "sniffOnStart" enabled (https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/configuration.html), since it seemed like a good idea in general. But turning it off fixed my issue. I suppose I'll have to either explicitly list my instances or put them behind a load balancer.

[wedgemartin]

Just wanted to add: this bug sucks. Seeing it with latest stable ( 5.4.0 ) Restarted it several times, and just running tests against it, I hit it 99.9% of the time.

[ess-avirank]

Is there any fix by now? I've tried using the global agent, agentkeepalive, turning off sniffing, and ended up where I started...

[n3ps]

For dev purposes only, mine was resolved with http.cors.enabled and http.cors.allow-origin

[calbear47]

I tried many solutions in this thread, but nothing was working for me. I was using a managed database service that utilized AWS EC2 service...the daas provided me two connections strings. If I didn't include both, then the connection would always die. So configured like this:

var client = new elasticsearch.Client({
    hosts: [
        'https://<user>:<password>@<host>:17132/',
        'https://<user>:<password>@<host>:17132/',
    ]
});

[TakeruTakahashi]

I have struggled against the errors in concurrent requests, but settting maxSockets to my client resolved it.

[chriswhong]

@TakeruTakahashi what did you set maxSockets to?

[silvae86]

I am implementing my own request queue using simple GET and PUT requests so I can handle connection drops to the ElasticSearch server gracefully. Too many hours (days!) spent trying to get something like this working sounds ridiculous in such a widely used, de-facto library. Connection pooling should be just that, pooling that abstracts connections from the client, instead of crashing left and right.

[fishcharlie]

Just for reference, this happened to me when I was trying to use an invalid character in the password field, but putting the url as http://username:password@host:9200. Having a ( in that password I think is what was causing problems.

[justinmchase]

I tried a couple of things here and was having the same issue at startup. It seemed to clear itself after a few minutes but that initial load was having this error.

I followed @toiye 's advice to set sniffOnStart: false and that seemed to clear it up. I realized that I only have a single instance at this point and so it was possible that sniffing causes an issue when you only have a single instance.

I'm not really sure but that may be a clue.

[stale[bot]]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours. It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy

[justinmchase]

For @stale bot's sake, i do think this is a legit issue. At the very least:

1. Sniffing for nodes in a non-clustered server causes a failure to connect to the single instance for several minutes on startup.

[delvedor]

Hello @justinmchase, thank you for writing! I missed this one, I'm sorry! How are you running Elasticsearch? Sniffing is a powerful tool, but it relies on the _nodes/_all/http endpoint, which means that if your Elasticsearch instance lives in a different network, the client will start using a wrong IP address.

For example, if you are spinning up Elasticsearch with docker, the sniffing will get the internal IP address (the one of the docker network) and therefore you will get a "no living connections" error.

I'm working on a blog post to explain better this situation. In the new client (which we will release very soon), sniffing will not work automatically if the cluster has just one node.

[justinmchase]

Yeah, in production we have multiple nodes and it is working fine but when testing locally I am just using a stock ES docker container and connecting from a different container in the same docker network. I was just noticing that when I first started up the whole network there was a period of time where the client was unable to connect. Eventually it worked itself out but I had to wait what seemed like several minutes before it would stop having errors. Setting sniffOnStart: false when developing locally totally resolved it for me and the client is immediately available.

[delvedor]

Thank you for sharing your experience! If you are using the correct ip inside docker there shouldn't be problems, maybe it's related to the docker network itself. If you want you can create a small repository to reproduce the issue, and I'll investigate more.

[justinmchase]

I haven't tried out the new client, I would be satisfied if simply updating to that resolved the issue. Otherwise I will do that when I get a chance. If you would prefer to close this and I file a new issue for that specifically then that's fine with me too, thanks for jumping in!

[delvedor]

If you would prefer to close this and I file a new issue for that specifically

Yes, please! Can you open it in elastic/elasticsearch-js-legacy? Please also add an example to reproduce the issue.

---

Thank you, everyone, for participating, I'm going to close this issue, please read this comment as the issue might be caused by this! If not, please fill a new issue in elastic/elasticsearch-js-legacy with an example to reproduce or in a few days, try the new client ;)

[design1online]

I followed @toiye 's advice to set sniffOnStart: false and that seemed to clear it up. I realized that I only have a single instance at this point and so it was possible that sniffing causes an issue when you only have a single instance.

This fixed my issue.