error=":excon is not registered on Faraday::Adapter"

[Sebov1c]

(check apply)

• [x] read the contribution guideline
• [ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

When I try to read a fluent.conf containing an elastic output the following error occurs:

fluentd-elastic-fluentd-1  | 2022-09-04 13:51:41 +0000 [info]: adding filter pattern="docker/**" type="parser"
fluentd-elastic-fluentd-1  | 2022-09-04 13:51:41 +0000 [info]: adding match pattern="docker/**" type="copy"
fluentd-elastic-fluentd-1  | 2022-09-04 13:51:42 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter"
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.5.2/lib/faraday/middleware_registry.rb:57:in `lookup_middleware'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.5.2/lib/faraday/rack_builder.rb:113:in `adapter'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/lib/ruby/3.1.0/forwardable.rb:238:in `adapter'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:600:in `block in client'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.5.2/lib/faraday/connection.rb:91:in `initialize'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.5.2/lib/faraday.rb:98:in `new'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.5.2/lib/faraday.rb:98:in `new'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/http/faraday.rb:65:in `__build_connection'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:169:in `block in __connections_from_host'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:161:in `map'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:161:in `__connections_from_host'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:149:in `__build_connections'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/elastic-transport-8.1.0/lib/elastic/transport/transport/base.rb:60:in `initialize'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `new'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `client'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:39:in `host_unreachable_exceptions'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:57:in `rescue in retry_operate'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:55:in `retry_operate'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin.rb:187:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin/multi_output.rb:99:in `each'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin/multi_output.rb:99:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin/out_copy.rb:39:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/plugin.rb:187:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/agent.rb:132:in `add_match'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/agent.rb:74:in `block in configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `each'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:149:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/engine.rb:105:in `configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/engine.rb:80:in `run_configure'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:763:in `block in run_worker'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:1037:in `main_process'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:755:in `run_worker'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/lib/fluent/command/fluentd.rb:381:in `<top (required)>'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.2/bin/fluentd:15:in `<top (required)>'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `load'
fluentd-elastic-fluentd-1  |   2022-09-04 13:51:42 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `<main>'
fluentd-elastic-fluentd-1  | 2022-09-04 13:51:42 +0000 [error]: Worker 0 finished unexpectedly with status 1

Steps to replicate

Dockerfile

FROM fluent/fluentd:v1.15.2-debian-1.0
USER root
RUN gem install fluent-plugin-elasticsearch --no-document --version 5.2.3 \
    && gem sources --clear-all \
    && apt-get purge -y --auto-remove \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
USER fluent

fluent.conf

<source>
  @type forward
</source>

<filter docker/**>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  <parse>
    @type json
  </parse>
</filter>

<match docker/**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    user elastic
    password "#{ENV['ELASTIC_PASSWORD']}"
    scheme https
    ssl_verify false
    validate_client_version true
    logstash_format true
    logstash_prefix fluentd
    include_tag_key true
  </store>

  <store>
    @type stdout
  </store>
</match>

version: '3'
services:
  fluentd:
    build: ./fluentd
    environment:
      - ELASTIC_USER=${ELASTIC_USER}
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    volumes:
    - ./fluentd/conf:/fluentd/etc/
    ports:
      - "127.0.0.1:24224:24224"
      - "127.0.0.1:24224:24224/udp"

Expected Behavior

Fluentd does not produce an error when reading a config containing elastic output plugin:

fluentd-elastic-fluentd-1  | 2022-09-04 14:16:03 +0000 [info]: adding match pattern="docker/**" type="stdout"
fluentd-elastic-fluentd-1  | 2022-09-04 14:16:03 +0000 [info]: adding source type="forward"
fluentd-elastic-fluentd-1  | 2022-09-04 14:16:03 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
fluentd-elastic-fluentd-1  | 2022-09-04 14:16:03 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
fluentd-elastic-fluentd-1  | 2022-09-04 14:16:03 +0000 [info]: #0 fluentd worker is now running worker=0

The OpenSearch plugin had a similar issue. Maybe this helps:

• https://github.com/fluent/fluent-plugin-opensearch/issues/70
• https://github.com/fluent/fluent-plugin-opensearch/pull/71

Using Fluentd and ES plugin versions

• OS version: GNU/Linux 5.15.60
• Docker: Docker engine 20.10.17
• Fluentd v0.12 or v0.14/v1.0
  • paste result of fluentd --version or td-agent --version
• ES plugin 3.x.y/2.x.y or 1.x.y
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
• ES version (optional)
• ES template(s) (optional)

FROM fluent/fluentd:v1.15.2-debian-1.0
RUN gem install fluent-plugin-elasticsearch --no-document --version 5.2.3 \

[josevromo]

I'm having this issue too. I've tried with different versions but so far every 5.x version has this problem.

Tried with 4.x versions and the error becomes:

fluentd_1   | /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
fluentd_1   |   from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-4.3.3/lib/fluent/plugin/elasticsearch_fallback_selector.rb:1:in `<top (required)>'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-4.3.3/lib/fluent/plugin/out_elasticsearch.rb:30:in `require_relative'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-4.3.3/lib/fluent/plugin/out_elasticsearch.rb:30:in `<top (required)>'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-4.3.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:2:in `require_relative'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-4.3.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:2:in `<top (required)>'
fluentd_1   |   from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd_1   |   from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/registry.rb:103:in `block in search'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/registry.rb:100:in `each'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/registry.rb:100:in `search'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/registry.rb:44:in `lookup'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin.rb:169:in `new_impl'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin.rb:114:in `new_output'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:130:in `add_match'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:74:in `block in configure'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `each'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/agent.rb:64:in `configure'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:149:in `configure'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:105:in `configure'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:80:in `run_configure'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:712:in `run_supervisor'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/command/fluentd.rb:361:in `<top (required)>'
fluentd_1   |   from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd_1   |   from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd_1   |   from /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/bin/fluentd:15:in `<top (required)>'
fluentd_1   |   from /usr/local/bundle/bin/fluentd:23:in `load'
fluentd_1   |   from /usr/local/bundle/bin/fluentd:23:in `<main>'

[j4dream]

Same issue.

[j4dream]

I installed faraday 1.10.0 and it seems to work RUN gem install faraday -v 1.10.0

[Sebov1c]

OK, so here is the temporary fix while this repository is not updated:

FROM fluent/fluentd:v1.15.2-debian-1.0
USER root
RUN gem install fluent-plugin-elasticsearch --no-document --version 5.2.3 \
    && gem uninstall --ignore-dependencies faraday \
    && gem install faraday --no-document --version 1.10.0 \
    && gem sources --clear-all \
    && apt-get purge -y --auto-remove \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
USER fluent

[yuval-gvs]

Seems to be the same issue as https://github.com/fluent/fluent-plugin-opensearch/issues/70 resulting from Faraday 2.x not including excon

[pferioli]

same workaround as above for alpine image

FROM fluent/fluentd:edge
USER root
RUN apk add --no-cache --update --virtual .build-deps build-base ruby-dev \
        && gem install fluent-plugin-elasticsearch --no-document --version 5.2.3 \
        && gem uninstall --ignore-dependencies faraday \
        && gem install faraday --no-document --version 1.10.0 \
        && gem sources --clear-all \
        && apk del .build-deps \
        && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

[linganmin]

Same issue.

[cosmo0920]

Thanks for your report. I'd released the patch for this issue as v5.2.4.

[raytung]

So I did a bit of digging on this issue, this is caused by Faraday gem moving its http backend adaptors out into their own respective gems. So to fix this issue, you'll need to install https://github.com/lostisland/faraday-excon, and run require faraday/excon, but seems like there's a bug with Faraday with Ruby 2.7 https://github.com/lostisland/faraday/issues/1444, and this is where I got up to. I'll pick up the further investigation once I have more time https://github.com/uken/fluent-plugin-elasticsearch/pull/1012

[cosmo0920]

but seems like there's a bug with Faraday with Ruby 2.7 lostisland/faraday#1444, and this is where I got up to. I'll pick up the further investigation once I have more time #1012

Ruby 2.7 is reached to EOL. No need to take care of.