search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
48 stars 3.5k forks source link

ElasticSearch ouput plugin is not compatible with other distributions starting from 7.16.0 #13506

Closed rbailly-talend closed 2 years ago

rbailly-talend commented 2 years ago

Logstash information:

Please include the following information:

  1. Logstash version: 7.16.0
  2. Logstash installation source: docker
  3. How is Logstash being run: kubernetes

Plugins installed: (bin/logstash-plugin list --verbose)

Description of the problem including expected versus actual behavior:

Since release 7.16.0 a change was introduce to perform some preflight checks on ElasticSearch output plugin. Those preflight checks are failing for some distributions of ElasticSearch, breaking the compatibility with OpenDistro / OpenSearch solutions. Reference to the breaking change: PR#1026.

Provide logs (if relevant):

Log captured from Logstash main pipeline when using an OpenDistro ElasticSearch cluster:

{
    "level": "ERROR",
    "loggerName": "logstash.javapipeline",
    "timeMillis": 1639401270427,
    "thread": "[main]-pipeline-manager",
    "logEvent": {
        "message": "Pipeline error",
        "pipeline_id": "main",
        "exception": {
            "metaClass": {
                "metaClass": {
                    "metaClass": {
                        "exception": "Could not connect to a compatible version of Elasticsearch",
                        "backtrace": [
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'",
                            "org/jruby/RubyHash.java:1415:in `each'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `start'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `build_pool'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'",
                            "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register'",
                            "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'",
                            "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'",
                            "org/jruby/RubyArray.java:1821:in `each'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'",
                            "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"
                        ],
                        "thread": "#<Thread:0x5567b83f@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:53 run>"
                    }
                }
            }
        }
    }
}

Result of the GET / query:

{
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-05-21T20:25:46.519671Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

As you can see the build_flavor is not compatible, for OpenSearc hthe tagline won't be compatible either

alexander-marquardt commented 2 years ago

Hi @rbailly-talend, sorry to heard that you are having issues with the Logstash Elasticsearch output. It sounds like you may be looking for the Amazon Elasticsearch output or the Opensearch output.

You can find additional context in: https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1018