search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
106 stars 80 forks source link

[BUG] Could not connect to cluster #73

Closed wkruse closed 3 years ago

wkruse commented 3 years ago

Describe the bug

We are getting Could not connect to cluster errors and Restored connection to OpenSearch instance warnings every 5 seconds.

2021/09/17 15:29:14 Setting 'path.config' from environment.
2021/09/17 15:29:14 Setting 'path.dead_letter_queue' from environment.
2021/09/17 15:29:14 Setting 'http.port' from environment.
2021/09/17 15:29:14 Setting 'http.host' from environment.
2021/09/17 15:29:14 Setting 'dead_letter_queue.enable' from environment.
2021/09/17 15:29:14 Setting 'pipeline.ecs_compatibility' from environment.
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2021-09-17T15:29:27,268][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2021-09-17T15:29:27,275][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-09-17T15:29:27,293][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2021-09-17T15:29:27,301][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2021-09-17T15:29:27,534][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-09-17T15:29:27,554][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"119a44c1-133b-4742-bdc9-b28ebc20f311", :path=>"/usr/share/logstash/data/uuid"}
[2021-09-17T15:29:28,051][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9301}
[2021-09-17T15:29:29,384][INFO ][org.reflections.Reflections] Reflections took 34 ms to scan 1 urls, producing 24 keys and 48 values
[2021-09-17T15:29:30,686][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["//opensearch:9450"]}
[2021-09-17T15:29:30,990][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://opensearch:9450/]}}
[2021-09-17T15:29:31,125][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:31,165][INFO ][logstash.outputs.opensearch][main] Cluster version determined (7.10.2) {:version=>7}
[2021-09-17T15:29:31,175][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:29:31,210][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"No Available connections", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::NoConnectionAvailableError, :backtrace=>["/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:349:in `with_connection'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:260:in `perform_request'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:268:in `block in get'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client.rb:208:in `get'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/plugin_mixins/opensearch/common.rb:90:in `discover_cluster_uuid'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch.rb:247:in `finish_register'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch.rb:225:in `block in register'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2021-09-17T15:29:31,215][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>7, :ecs_compatibility=>:disabled}
[2021-09-17T15:29:31,238][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"No Available connections", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::NoConnectionAvailableError, :backtrace=>["/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:349:in `with_connection'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:260:in `perform_request'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/pool.rb:268:in `block in Pool'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client.rb:388:in `exists?'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client.rb:393:in `template_exists?'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client.rb:89:in `template_install'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/template_manager.rb:37:in `install'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch/template_manager.rb:25:in `install_template'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch.rb:413:in `install_template'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch.rb:248:in `finish_register'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/outputs/opensearch.rb:225:in `block in register'", "/usr/share/logstash/vendor/local_gems/217134f0/logstash-output-opensearch/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2021-09-17T15:29:31,279][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>32, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>4000, "pipeline.sources"=>["/etc/logstash/config.json"], :thread=>"#<Thread:0x6bbabd1d run>"}
[2021-09-17T15:29:33,695][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>2.41}
[2021-09-17T15:29:33,741][INFO ][logstash.inputs.redis    ][main] Registering Redis {:identity=>"redis://@redis:6379/0 list:logstash"}
[2021-09-17T15:29:33,777][INFO ][logstash.inputs.redis    ][main] Registering Redis {:identity=>"redis://@redis:6379/0 list:logstash"}
[2021-09-17T15:29:33,777][INFO ][logstash.inputs.redis    ][main] Registering Redis {:identity=>"redis://@redis:6379/0 list:logstash"}
[2021-09-17T15:29:33,778][INFO ][logstash.inputs.redis    ][main] Registering Redis {:identity=>"redis://@redis:6379/0 list:logstash"}
[2021-09-17T15:29:33,786][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-09-17T15:29:33,885][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-09-17T15:29:36,190][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:36,197][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:29:41,201][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:41,207][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:29:46,212][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:46,220][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:29:51,225][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:51,230][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:29:56,236][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:29:56,240][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:30:01,244][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:30:01,248][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:30:06,252][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:30:06,256][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}
[2021-09-17T15:30:11,260][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"http://opensearch:9450/"}
[2021-09-17T15:30:11,264][ERROR][logstash.outputs.opensearch][main] Could not connect to cluster {:url=>"http://opensearch:9450/", :distribution=>nil, :major_version=>7}

Running tcpdump we see that the health checks succeed.

HEAD / HTTP/1.1
Connection: Keep-Alive
Content-Type: application/json
Host: opensearch:9450
User-Agent: Manticore 0.7.0
Accept-Encoding: gzip,deflate

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 534

GET / HTTP/1.1
Connection: Keep-Alive
Content-Type: application/json
Content-Length: 0
Host: opensearch:9450
User-Agent: Manticore 0.7.0
Accept-Encoding: gzip,deflate

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 330

{
  "name" : "opensearch-2",
  "cluster_name" : "opensearch-dev",
  "cluster_uuid" : "CIbySD6EScKrd29iEjNpfg",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "tar",
    "build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date" : "2021-07-02T23:22:21.383695Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

OpenSearch output plugin config

output {
  opensearch {
    hosts => ["opensearch:9450"]
    index => "logstash-%{[system]}-%{+YYYY.MM.dd}"
  }
}

To Reproduce Steps to reproduce the behavior:

  1. Start 3 node OpenSearch cluster in Kubernetes
  2. Start Logstash 7.13.x with OpenSearch output plugin

Expected behavior No Could not connect to cluster errors.

Host/Environment (please complete the following information):

Additional context opensearch:9450 is a service in Kubernetes in front of 3 OpenSearch nodes. We also tried the plugin with a list of our 3 nodes (both internal Kubernetes StatefulSet names and external host names) and were still getting the errors.

Logstash OSS 7.10.2 with Elasticsearch output plugin works without problems.

wkruse commented 3 years ago

We also tried setting validate_after_inactivity to 0 without success.

output {
  opensearch {
    hosts => ["opensearch:9450"]
    validate_after_inactivity => 0
    index => "logstash-%{[system]}-%{+YYYY.MM.dd}"
  }
}

https://www.elastic.co/guide/en/logstash/master/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-validate_after_inactivity

VijayanB commented 3 years ago

@wkruse Thanks for reporting this bug. I see that you are running opensearch in backward compatible mode from version number. It is known bug and fix is merged in 1.1. we will release it on Sept 21.

VijayanB commented 3 years ago

@wkruse we release 1.1.0 with fix. Can you please uninstall and install plugin and let us know did that solve your problem?

bin/logstash-plugin uninstall logstash-output-opensearch
bin/logstash-plugin install logstash-output-opensearch
wkruse commented 3 years ago

@VijayanB 1.1.0 solved it for us. Thank you!