search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.03k stars 24.83k forks source link

Intermittent errors connecting to ES cluster when using High Level Rest client #62127

Closed jdevilla20 closed 4 years ago

jdevilla20 commented 4 years ago

Elasticsearch version (bin/elasticsearch --version): 7.6.2

Plugins installed: []

JVM version (java -version): sh-4.2# /usr/share/elasticsearch/jdk/bin/java -version openjdk version "13.0.2" 2020-01-14 OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.2+8) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.2+8, mixed mode, sharing) OS version (uname -a if on a Unix-like system): Linux x86_64 x86_64 x86_64 GNU/Linux Description of the problem including expected versus actual behavior:

Steps to reproduce:

I am running a 3 node ES cluster on Kubernetes and performing about 80 upserts per minute. When connecting via the High Level Rest client, I see intermittent errors (see below). Other times, the updates work fine.

When I look at my CPU usage, the hot cpu is running at 15% usage.

I am not sure if the errors are with the cluster or with the Rest client.

Rest Client initialization:


  private RestHighLevelClient initClient(ElasticSearchStoreConfig config) throws IOException {
      String password = "foobar";

      final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
      credentialsProvider.setCredentials(AuthScope.ANY,
          new UsernamePasswordCredentials(config.getUserName(), password));

      RestClientBuilder builder = RestClient.builder(new HttpHost(config.getHostname(),
          config.getPort(), config.getElasticSearchScheme()))
          .setHttpClientConfigCallback(httpClientBuilder ->
              httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));

      return new RestHighLevelClient(builder);

  }

I am running the update as follows (there is a retry policy): 

      UpdateRequest updateRequest = mapper.update(key, data);
      UpdateResponse response = esClient.update(updateRequest, RequestOptions.DEFAULT);

Provide logs (if relevant):

[DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] PoolingNHttpClientConnectionManager: - Connection request failed
java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
    at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
    at java.lang.Thread.run(Thread.java:830) [?:?]
[DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] InternalHttpAsyncClient: - [exchange: 801] connection request failed
[DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - request [POST http://ingestion-es-http:9200/recordmetadata-fei/_update/1599066000000-1599069600000:testapp:ingestiontenant3:firewall.traffic?retry_on_conflict=5&timeout=1m] failed
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused
    at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71) ~[httpcore-4.4.12.jar!/:4.4.12]
    at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84) ~[httpcore-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:244) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0]
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1609) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1579) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1549) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
    at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:1016) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
    at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:182) ~[ingest-common-2.0.0.jar!/:?]
    at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:36) ~[ingest-common-2.0.0.jar!/:?]
    at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:58) ~[classes!/:?]
    at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:19) ~[classes!/:?]
    at com.paloaltonetworks.cortex.ingestion.beacon.BeaconWriter.run(BeaconWriter.java:65) ~[ingest-common-2.0.0.jar!/:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
    at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
    ... 1 more
[DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - updated [[host=http://ingestion-es-http:9200]] already in blacklist
cjcenizal commented 4 years ago

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds of things to the forums. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.

There's an active community in the forums that should be able to help get an answer to your question. As such, I hope you don't mind that I close this.