search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Using fluent-plugin-elasticsearch `v.4.1.2` and above does not work with Elasticsearch `v.7.7.1` #854

Closed adnanrahic closed 3 years ago

adnanrahic commented 3 years ago

(check apply)

Problem

While following this guide, log shipping does not work and ends up with this error:

[error]: #0 unexpected error error_class=NoMethodError error="undefined method `dig' for #<String:0x0000557b648ef9d8>"

Steps to replicate

I used this config:

# Switch to debug if you need to debug
<system>
  log_level debug
</system>

# get logs from syslog
<source>
  @type syslog
  port 42185
  tag syslog
</source>

# get logs from fluent-logger, fluent-cat or other Fluentd instances
<source>
  @type forward
</source>

<filter **>
  @type stdout
</filter>

<match syslog.**>
  @type elasticsearch
  host logsene-receiver.sematext.com
  # for EU
  # host logsene-receiver.eu.sematext.com
  port 443
  scheme https
  index_name <LOGS_TOKEN>
  <buffer>
    @type file
    path /tmp/fluent/es-buffer/es.all.*.buffer
    chunk_limit_size 250k
    flush_interval 50s
    flush_thread_count 1
    retry_limit 5
    retry_wait 60
  </buffer>
</match>

The systemd service starts, but log shipping does not work, and shows this error in /var/log/td-agent/td-agent.log:

[error]: #0 unexpected error error_class=NoMethodError error="undefined method `dig' for #<String:0x0000557b648ef9d8>"

Expected Behavior or What you need to ask

Log shipping should work.

After testing with other versions of both Elasticsearch and fluent-plugin-elasticsearch I noticed this issue it NOT present with v.4.1.1 of fluent-plugin-elasticsearch and Elasticsearch v.7.7.1.

The issue is NOT present with Elasticsearch v.7.4.0 and fluent-plugin-elasticsearch v.4.1.2 and above.

Do you know why is this happening? :thinking:

Using Fluentd and ES plugin versions

Not working:

Working:

OR

cosmo0920 commented 3 years ago

What is the result to execute the following commands?

require 'elasticsearch'

client = Elasticsearch::Client.new(host: "logsene-receiver.sematext.com:443")
puts client.info
puts client.info.class
cosmo0920 commented 3 years ago

Patch is here: https://github.com/uken/fluent-plugin-elasticsearch/pull/855

adnanrahic commented 3 years ago

Hi @cosmo0920! Thanks for looking into this. Is there anything else you'd need from me? Or, will patch #855 be merged?

cosmo0920 commented 3 years ago

855 is merged. It will be released soon.

cosmo0920 commented 3 years ago

855 is released as v4.3.3.