fluentd not communication with elasticsearch host_unreachable_exceptions

ambigus9 commented 2 years ago

Problem

Can't connect Fluentd with Elasticsearch

Steps to replicate

Create Dockerfile with following content:

FROM fluent/fluentd:v1.12.0-debian-1.0

USER root
ADD conf/fluent.conf /etc/fluent/
ADD conf/parser_custom_parser.rb /etc/fluent/
ADD conf/parser_custom_parser.rb /etc/fluent/plugin/
ADD conf/parser_custom_parser.rb /lib/fluent/plugin/
ADD conf/formatter_custom_formatter.rb /etc/fluent/
ADD conf/formatter_custom_formatter.rb /etc/fluent/plugin/
ADD conf/formatter_custom_formatter.rb /lib/fluent/plugin/
ADD ca.pem /etc/fluent/certs/ca.pem
ENV HTTP_PROXY="http://100.126.0.150:3188"
ENV HTTPS_PROXY="http://100.126.0.150:3188"
RUN apt-get update && apt-get install -y curl && apt-get install -y telnet
RUN echo "source 'https://mirrors.tuna.tsinghua.edu.cn/rubygems/'" > Gemfile$
RUN echo "100.126.19.110 kafka" >> /etc/hosts
RUN echo "100.126.19.55 kafka02" >> /etc/hosts
RUN gem install elasticsearch -v 7.16.2
RUN gem install fluent-plugin-elasticsearch  -v 5.0.5 --no-document
RUN gem install fluent-plugin-kafka -v 0.12.3 --no-document
ENV HTTP_PROXY=
ENV HTTPS_PROXY=
ENV http_proxy=
ENV https_proxy=
RUN echo "unset http_proxy" >> /root/.bashrc
RUN echo "unset https_proxy" >> /root/.bashrc
RUN echo "unset HTTP_PROXY" >> /root/.bashrc
RUN echo "unset HTTPS_PROXY" >> /root/.bashrc

CMD ["fluentd"]

Create the file fluentd.conf

# fluentd/conf/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
  tag logs.test
</source>

<source>
  @type kafka
  brokers kafka:9092,kafka02:9092
  format text
  <topic>
    topic app_gw_uat
  </topic>
</source>

<filter *.**>
  @type parser
  key_name message
  <parse>
    @type custom_parser
  </parse>
</filter>

<match *.**>
  @type copy
  <store>
    @type stdout
  </store>
</match>

<match *.**>
  @type elasticsearch
  @log_level debug
  host 100.123.251.89
  port 9200
  scheme https
  ssl_verify false
  #ssl_version TLSv1_2
  ca_file /etc/fluent/certs/ca.pem
  client_cert /etc/fluent/certs/ca.pem
  user MY_USER
  password MY_PASSWORD
  logstash_format true
  #logstash_prefix fluentd-
  index_name fluentd-01
  type_name fluentd-01
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  tag

Expected Behavior or What you need to ask

Correctly index data from Fluentd in Elasticsearch.

Using Fluentd and ES plugin versions

OS version: Mac OS Monterrey 12.3.1
Docker version: 20.10.14, build a22408
Fluentd v1.12.0
ES plugin 5.0.5
ES version 7.16.3

To expand info here are the logs:

luentd_1  | 2022-04-18 15:25:47 +0000 [debug]: #0 Need substitution: false
fluentd_1  | 2022-04-18 15:25:47 +0000 [debug]: #0 'host_placeholder 100.123.251.89' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: 100.123.251.89' doesn't have tag placeholder
fluentd_1  | 2022-04-18 15:25:52 +0000 [error]: #0 [Faraday::TimeoutError] read timeout reached {:host=>"100.123.251.89", :port=>9200, :scheme=>"https", :user=>"inspiraefk", :password=><REDACTED>, :protocol=>"https"}
fluentd_1  | 2022-04-18 15:25:52 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `host_unreachable_exceptions' for #<Elasticsearch::Transport::Client:0x00007efdbbf513c8>"
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:41:in `rescue in retry_operate'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:39:in `retry_operate'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin.rb:178:in `configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:132:in `add_match'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:74:in `block in configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:64:in `each'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:64:in `configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/root_agent.rb:146:in `configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/engine.rb:105:in `configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/engine.rb:80:in `run_configure'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/supervisor.rb:692:in `block in run_worker'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/supervisor.rb:943:in `main_process'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/supervisor.rb:684:in `run_worker'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/command/fluentd.rb:361:in `<top (required)>'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.12.0/bin/fluentd:8:in `<top (required)>'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
fluentd_1  |   2022-04-18 15:25:52 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
fluentd_1  | 2022-04-18 15:25:52 +0000 [info]: Worker 0 finished unexpectedly with status 1
fluentd_1  | 2022-04-18 15:25:52 +0000 [info]: adding filter pattern="*.**" type="parser"
fluentd_1  | 2022-04-18 15:25:52 +0000 [info]: adding match pattern="*.**" type="copy"
fluentd_1  | 2022-04-18 15:25:52 +0000 [info]: adding match pattern="*.**" type="elasticsearch"
fluentd_1  | 2022-04-18 15:25:53 +0000 [debug]: #0 'host 100.123.251.89' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: 100.123.251.89' doesn't have tag placeholder

cosmo0920 commented 2 years ago

Your using ES plugin seems to be old:

RUN gem install fluent-plugin-elasticsearch -v 5.0.5 --no-document

Could you use the latest version of ES plugin(5.2.2)?

ambigus9 commented 2 years ago

@cosmo0920 Thanks!! We found that was a problem with the node and OS where we was working on. Thanks!

maverick2808 commented 1 year ago

@ambigus9 : How exactly was the issue fixed & what were the problems identified with the node and OS ? Thanks.

uken / fluent-plugin-elasticsearch