Elastic 8.x - NoMethodError error="undefined method `ilm' for <Elasticsearch::Transport::Client> in 5.0.1

[lancegliser]

• [X] read the contribution guideline
• [X] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

I note that Elastic stack 8.x support was added in #949 . Something appears to have changed in the Index Lifecycle Management apis:

2022-03-08 20:10:37 +0000 [info]: #0 Installing ILM policy: {"policy"=>{"phases"=>{"hot"=>{"min_age"=>"0ms", "actions"=>{"rollover"=>{"max_age"=>"3d", "max_size"=>"50mb"}, "set_priority"=>{"priority"=>100}}}, "delete"=>{"min_age"=>"30d", "actions"=>{"delete"=>{"delete_searchable_snapshot"=>true}}}}}}
2022-03-08 20:10:37 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `ilm' for #<Elasticsearch::Transport::Client:0x0000563da5aede48>"
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb:60:in `ilm_policy_put'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb:31:in `create_ilm_policy'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb:7:in `setup_ilm'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/elasticsearch_index_template.rb:188:in `create_rollover_alias'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/out_elasticsearch.rb:1029:in `block in template_installation_actual'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/elasticsearch_index_template.rb:40:in `retry_operate'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/out_elasticsearch.rb:1020:in `template_installation_actual'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-elasticsearch-5.0.1/lib/fluent/plugin/out_elasticsearch.rb:262:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin.rb:164:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/multi_output.rb:74:in `block in configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/multi_output.rb:63:in `each'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/multi_output.rb:63:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/out_copy.rb:36:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin.rb:164:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/agent.rb:130:in `add_match'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/agent.rb:72:in `block in configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/agent.rb:64:in `each'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/agent.rb:64:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/root_agent.rb:112:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/engine.rb:131:in `configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/engine.rb:96:in `run_configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:795:in `run_configure'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:548:in `block in run_worker'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:724:in `main_process'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:544:in `run_worker'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/bin/fluentd:8:in `<top (required)>'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
  2022-03-08 20:10:37 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'

...

Steps to replicate

# Send it to Elastic
<match **>
  @type copy

  <store>
    # https://github.com/uken/fluent-plugin-elasticsearch
    @type elasticsearch
    # You need debug if you want to know why your data is rejected if you start seeing
    # Host
    scheme https
    host es01
    port 9200
    ssl_verify false
# Never did seem to get these working based on https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-stack-docker.html#get-started-docker-tls
#     ca_file /etc/fluentd/certs/ca/ca.crt
#     client_cert /etc/fluentd/certs/ca/ca.crt
#     client_key /etc/fluentd/certs/ca/ca.key
    # Authentication
    user elastic
    password *****
    # General
    include_tag_key true
    include_timestamp true
    flush_interval 1s
    suppress_type_name true
    # Index
    index_name fluent-ecs
    index_date_pattern ""
    # Index Lifecycle Management
    enable_ilm true
    rollover_index true
    ilm_policy_id fluent-ecs
    ilm_policy_overwrite true
    ilm_policy {"policy": {"phases": {"hot": {"min_age": "0ms","actions": {"rollover": {"max_age": "3d","max_size": "50mb"},"set_priority": {"priority": 100}}},"delete": {"min_age": "30d","actions": {"delete": {"delete_searchable_snapshot": true}}}}}}
    # Template
    template_name fluentd-ecs
    template_file /etc/fluentd/ecs-template.json
  </store>
</match>

Expected Behavior or What you need to ask

Expected ILM to be installed as it use to.

Using Fluentd and ES plugin versions

• OS version: Mac
• Docker compose through Docker desktop
• Fluentd v0.12 or v0.14/v1.0

  2022-03-08 20:10:33 +0000 [info]: starting fluentd-1.3.2 pid=8 ruby="2.5.2"
  2022-03-08 20:10:33 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]

• ES plugin 3.x.y/2.x.y or 1.x.y

  2022-03-08 20:10:33 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.1'
  2022-03-08 20:10:33 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
  2022-03-08 20:10:33 +0000 [info]: gem 'fluentd' version '1.3.2'

• ES version 8.0.1
• ES template(s): Elastic Common Schema

[cosmo0920]

The old version of ES plugin does not support Elasticsearch 8 or later. Please consider to use v5.2.0 or later. Thanks.

[lancegliser]

Thanks @cosmo0920 I missed that part in testing. I'll get it moving.

[mokhos]

Same issue here. The difference is that I'm using bitnami/fluentd:v1.14.5-debian-10-r1 which contains fluent-plugin-elasticsearch-5.1.5, but still getting this undefined method:

2022-03-30 10:30:48 +0000 [info]: adding match pattern="**" type="elasticsearch"
2022-03-30 10:30:48 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `+' for nil:NilClass"
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/excon-0.91.0/lib/excon/connection.rb:247:in `request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `block in call'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/faraday-1.9.3/lib/faraday/adapter.rb:50:in `connection'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `call'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/faraday-1.9.3/lib/faraday/rack_builder.rb:154:in `build_response'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/faraday-1.9.3/lib/faraday/connection.rb:516:in `run_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.17.0/lib/elasticsearch/transport/transport/http/faraday.rb:50:in `block in perform_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.17.0/lib/elasticsearch/transport/transport/base.rb:289:in `perform_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.17.0/lib/elasticsearch/transport/transport/http/faraday.rb:36:in `perform_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.17.0/lib/elasticsearch/transport/client.rb:197:in `perform_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-7.17.0/lib/elasticsearch.rb:93:in `elasticsearch_validation_request'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-7.17.0/lib/elasticsearch.rb:51:in `verify_elasticsearch'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-7.17.0/lib/elasticsearch.rb:40:in `method_missing'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-api-7.17.0/lib/elasticsearch/api/actions/info.rb:37:in `info'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:490:in `block in handle_last_seen_es_major_version'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/elasticsearch_index_template.rb:52:in `retry_operate'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/plugin.rb:187:in `configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:132:in `add_match'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:74:in `block in configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `each'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/agent.rb:64:in `configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/root_agent.rb:149:in `configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/engine.rb:105:in `configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/engine.rb:80:in `run_configure'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:716:in `block in run_worker'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:968:in `main_process'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/supervisor.rb:708:in `run_worker'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/ruby/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/ruby/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.14.5/bin/fluentd:15:in `<top (required)>'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/bin/fluentd:23:in `load'
  2022-03-30 10:30:48 +0000 [error]: #0 /opt/bitnami/fluentd/bin/fluentd:23:in `<main>'
2022-03-30 10:30:48 +0000 [error]: Worker 0 finished unexpectedly with status 1

[cosmo0920]

@mokhos Please report your issue into bitnami’s repo. We are not familiar with bitnami’s image. Thanks.