search

uken / fluent-plugin-elasticsearch

Apache License 2.0
888 stars 309 forks source link

Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError) #1014

Open danfinn opened 1 year ago

danfinn commented 1 year ago

(check apply)

Problem

Recently started seeing this error when starting our fluentd pods. We did recently upgrade the elasticsearch gem from 7.17.7 to 8.6 but I'm nearly sure we weren't seeing this error after making that change.

2023-04-05 22:00:03 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-04-05 22:00:03 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/main.conf"
2023-04-05 22:00:03 +0000 [info]: gem 'fluentd' version '1.16.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.5'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-04-05 22:00:03 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-04-05 22:00:03 +0000 [info]: DeDot will recurse nested hashes and arrays
/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.1.0/rubygems/specification.rb:1419:in `activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:211:in `rescue in try_activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:204:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:153:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.1.0/rubygems/specification.rb:1419:in `activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:205:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:153:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- fluent/log-ext (LoadError)
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'

...

Steps to replicate

Here's our Dockerfile:

FROM fluent/fluentd:v1.15-debian-1

USER root
WORKDIR /home/fluent

#ENV PATH /fluentd/vendor/bundle/ruby/2.6.0/bin:$PATH
#ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.6.0
#ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.6.0
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install elasticsearch -v 8.6 \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem install fluent-plugin-multi-format-parser \
 && sudo gem install fluent-plugin-concat \
 && sudo gem install fluent-plugin-grok-parser \
 && sudo gem install fluent-plugin-prometheus \
 && sudo gem install fluent-plugin-json-in-json-2 \
 && sudo gem install fluent-plugin-record-modifier \
 && sudo gem install fluent-plugin-detect-exceptions \
 && sudo gem install fluent-plugin-rewrite-tag-filter \
 && sudo gem install fluent-plugin-dedot_filter \
 && sudo gem install fluent-plugin-kubernetes_metadata_filter \
 && sudo gem install ffi \
 && sudo gem install fluent-plugin-systemd \
 && sudo gem install fluent-plugin-parser-cri \
 && sudo gem install fluent-plugin-azure-storage-append-blob \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

ADD conf /fluentd/etc/
ADD plugins /fluentd/plugins/

COPY entrypoint.sh /fluentd/entrypoint.sh

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

EXPOSE 9292:9292

# Overwrite ENTRYPOINT to run fluentd as root for /var/log / /var/lib
ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]

Expected Behavior or What you need to ask

the pod to start without errors

Using Fluentd and ES plugin versions

I did also try using FROM fluent/fluentd:v1.16-debian-1 but there was no change there. I saw that there was a PR 3 days ago related to unpinning the version of faraday, perhaps that is related to this? https://github.com/uken/fluent-plugin-elasticsearch/pull/1012

danfinn commented 1 year ago

I tried to downgrade the elasticsearch gem to 7.17.7 which is what we were previously using and still get the same error on startup.

danfinn commented 1 year ago

Pinning the install of fluent-plugin-elasticsearch to 5.2.3 gets the pod running but now I get a new error:

2023-04-06 00:30:36 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter"
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/faraday-2.7.4/lib/faraday/middleware_registry.rb:57:in `lookup_middleware'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/faraday-2.7.4/lib/faraday/rack_builder.rb:113:in `adapter'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/3.1.0/forwardable.rb:238:in `adapter'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:600:in `block in client'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/faraday-2.7.4/lib/faraday/connection.rb:91:in `initialize'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/faraday-2.7.4/lib/faraday.rb:98:in `new'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/faraday-2.7.4/lib/faraday.rb:98:in `new'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:66:in `__build_connection'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:169:in `block in __connections_from_host'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:161:in `map'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:161:in `__connections_from_host'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:149:in `__build_connections'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:60:in `initialize'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `new'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:617:in `client'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:39:in `host_unreachable_exceptions'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:57:in `rescue in retry_operate'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/elasticsearch_index_template.rb:55:in `retry_operate'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.3/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:187:in `configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:132:in `add_match'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:616:in `block in run_worker'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:962:in `main_process'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:608:in `run_worker'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2023-04-06 00:30:36 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-06 00:30:36 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `load'
  2023-04-06 00:30:36 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `<main>'

This is now the same error as https://github.com/uken/fluent-plugin-elasticsearch/issues/984

raytung commented 1 year ago

I saw that there was a PR 3 days ago related to unpinning the version of faraday, perhaps that is related to this?

Hey @danfinn, my understanding is that while the PR is merged we haven't done a release yet. One thing to note is that I can see you are using fluent-plugin-azure-storage-append-blob which is now unsupported and uses Faraday v1.x so this will definitely cause a conflict when the author eventually release https://github.com/uken/fluent-plugin-elasticsearch/pull/1012

danfinn commented 1 year ago

Unsupported or unmaintained? I think those are 2 different things?

Do you know if https://github.com/elsesiy/fluent-plugin-azure-storage-append-blob-lts also uses faraday v1.x?

danfinn commented 1 year ago

It seems like the suggestion from #984 of installing faraday 1.10.0 has things working again.

raytung commented 1 year ago

Do you know if https://github.com/elsesiy/fluent-plugin-azure-storage-append-blob-lts also uses faraday v1.x?

@danfinn Unfortunately it does. https://github.com/Azure/azure-storage-ruby/blob/master/common/azure-storage-common.gemspec#L44. The problem actually lies in the 2nd layer of transitive dependency

fluent-plugin-azure-storage-append-blob -> azure-storage-blob -> azure-storage-common. Looks like there's an open issue on the Azure library on this https://github.com/Azure/azure-storage-ruby/issues/225

danfinn commented 1 year ago

The issue does not appear to be the fluent-plugin-azure-storage-append-blob gem. I removed that from my dockerfile and I still get the same error:

/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)

Here is my Dockerfile that generated this error:

FROM fluent/fluentd:v1.15-debian-1

USER root
WORKDIR /home/fluent

#ENV PATH /fluentd/vendor/bundle/ruby/2.6.0/bin:$PATH
#ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.6.0
#ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.6.0
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 # && sudo gem install faraday --no-document --version 1.10.0 \
 && sudo gem install elasticsearch -v 8.6 \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem install fluent-plugin-multi-format-parser \
 && sudo gem install fluent-plugin-concat \
 && sudo gem install fluent-plugin-grok-parser \
 && sudo gem install fluent-plugin-prometheus \
 && sudo gem install fluent-plugin-json-in-json-2 \
 && sudo gem install fluent-plugin-record-modifier \
 && sudo gem install fluent-plugin-detect-exceptions \
 && sudo gem install fluent-plugin-rewrite-tag-filter \
 && sudo gem install fluent-plugin-dedot_filter \
 && sudo gem install fluent-plugin-kubernetes_metadata_filter \
 && sudo gem install ffi \
 && sudo gem install fluent-plugin-systemd \
 && sudo gem install fluent-plugin-parser-cri \
 # && sudo gem install fluent-plugin-azure-storage-append-blob \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

ADD conf /fluentd/etc/
ADD plugins /fluentd/plugins/

COPY entrypoint.sh /fluentd/entrypoint.sh

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

EXPOSE 9292:9292

# Overwrite ENTRYPOINT to run fluentd as root for /var/log / /var/lib
ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]
cosmo0920 commented 1 year ago

Could you use bundle and Gemfile to manage your gem dependencies instead? This is because gem install always should use/fetch the latest gems. Bundler should follow the versions that are user specified.

cosmo0920 commented 1 year ago

Will be handled to publish this plugin as v5.3.0.

kenhys commented 1 year ago

v5.3.0 has been released so I hope your issue will be solved.

danfinn commented 1 year ago

ok, I will try building again with v5.3.0

brook-w commented 1 year ago

Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) WTF

brook-w commented 1 year ago

https://github.com/uken/fluent-plugin-elasticsearch/issues/1014#issuecomment-1502892321

brook-w commented 1 year ago

Unable to activate fluent-plugin-elasticsearch-5.3.0, because faraday-1.10.3 conflicts with faraday (>= 2.0.0), faraday-excon-1.1.0 conflicts with faraday-excon (>= 2.0.0) WTF

v5.3.0 has been released so I hope your issue will be solved.