search

uken / fluent-plugin-elasticsearch

Apache License 2.0
890 stars 310 forks source link

Crush with elasticsearch server 7.2 #951

Closed kvokka closed 2 years ago

kvokka commented 2 years ago

(check apply)

Problem

Elasticrearch 7.2 support vanished after the update to 5.2.0 from 5.1.5. Fluentd config was unchanged

logs

2022-02-23 04:21:59 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="stdout"
2022-02-23 04:21:59 +0000 [info]: adding match in @NORMAL pattern="**" type="elasticsearch"
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2022-02-23 04:22:00 +0000 [error]: unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not Elasticsearch and we do not support this unknown product."
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/elasticsearch-8.0.0/lib/elasticsearch.rb:115:in `verify_with_version_or_header'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/elasticsearch-8.0.0/lib/elasticsearch.rb:104:in `verify_elasticsearch'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/elasticsearch-8.0.0/lib/elasticsearch.rb:70:in `method_missing'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/elasticsearch-api-8.0.0/lib/elasticsearch/api/actions/info.rb:39:in `info'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.2.0/lib/fluent/plugin/out_elasticsearch.rb:495:in `detect_es_major_version'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.2.0/lib/fluent/plugin/out_elasticsearch.rb:487:in `block in handle_last_seen_es_major_version'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.2.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.2.0/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.2.0/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/plugin.rb:187:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:132:in `add_match'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:74:in `block in configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `each'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/label.rb:31:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/root_agent.rb:146:in `block in configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/root_agent.rb:146:in `each'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/root_agent.rb:146:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/engine.rb:105:in `configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/engine.rb:80:in `run_configure'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:716:in `block in run_worker'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:968:in `main_process'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:708:in `run_worker'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/ruby/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/ruby/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/gems/fluentd-1.14.5/bin/fluentd:15:in `<top (required)>'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/bin/fluentd:23:in `load'
  2022-02-23 04:22:00 +0000 [error]: /opt/bitnami/fluentd/bin/fluentd:23:in `<main>'

CONFIG

          <match **>
            @id elasticsearch
            @type elasticsearch
            @log_level info
            include_tag_key true
            host "{{ .Values.global.chart_name }}-es.{{ .Values.global.es_service.domain }}"
            port {{ .Values.global.es_service.httpPort }}
            type_name _doc
            logstash_format true
            reconnect_on_error true
            reload_on_failure true
            reload_connections false
            compression_level default_compression
            prefer_oj_serializer true
            http_backend typhoeus
            <buffer>
              @type file
              path /opt/bitnami/fluentd/logs/buffers/elasticsearch
              flush_mode interval
              retry_type exponential_backoff
              flush_thread_count 2
              flush_interval 5s
              retry_max_interval 10
              chunk_limit_size 32M
              queue_limit_length 8
              overflow_action drop_oldest_chunk
            </buffer>
          </match>

Using Fluentd and ES plugin versions

ppadron commented 2 years ago

Are you using OSS Elasticsearch? If so, downgrade the elasticsearch gem to 7.14:

gem install elasticsearch --version '<= 7.14'

With the 7.11 the release of Elasticsearch, we will no longer be releasing oss-only distributions, as stated in the FAQ on license change. The recommendation is to upgrade to the latest default distribution of Elasticsearch, which is free to use under Elastic License v2. Otherwise, you can pin the client version to <7.14.

Reference: https://github.com/elastic/elasticsearch-py/issues/1639#issuecomment-883319286

kvokka commented 2 years ago

Thank you for the advice. With the image https://github.com/bitnami/bitnami-docker-fluentd version 1.14.5-debian-10-r11 I can only rollback to elasticsearch:7.14.1 to make this run. But this hack is not supposed to be a default solution IMO.

Gemspec does not lock dependency versions.

Hope this issue will save a day for someone and will be happy if the maintainers of this gem either drop support of older versions or fix the gem or update the readme at least.

cosmo0920 commented 2 years ago

Gemspec does not lock dependency versions.

Hope this issue will save a day for someone and will be happy if the maintainers of this gem either drop support of older versions or fix the gem or update the readme at least.

We don't plan to lock gemspec versions for elasticsearch related gems. It is compatibility issue. Maybe ES plugin should catch Unsupported exceptions and report appropriate errors.