search

uken / fluent-plugin-elasticsearch

Apache License 2.0
888 stars 309 forks source link

Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError) #1013

Closed danfinn closed 1 year ago

danfinn commented 1 year ago

Recently started seeing this error when starting our fluentd pods. We did recently upgrade the elasticsearch gem from 7.17.7 to 8.6 but I'm nearly sure we weren't seeing this error after making that change.

2023-04-05 22:00:03 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-04-05 22:00:03 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/main.conf"
2023-04-05 22:00:03 +0000 [info]: gem 'fluentd' version '1.16.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.5'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-05 22:00:03 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-04-05 22:00:03 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-04-05 22:00:03 +0000 [info]: DeDot will recurse nested hashes and arrays
/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.1.0/rubygems/specification.rb:1419:in `activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:211:in `rescue in try_activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:204:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:153:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.5, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.1.0/rubygems/specification.rb:1419:in `activate'
    from /usr/local/lib/ruby/3.1.0/rubygems.rb:205:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:153:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- fluent/log-ext (LoadError)
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:100:in `search'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `each'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/local/lib/ruby/gems/3.1.0/gems/fluentd-1.16.0/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'

Here's our Dockerfile:

FROM fluent/fluentd:v1.15-debian-1

USER root
WORKDIR /home/fluent

#ENV PATH /fluentd/vendor/bundle/ruby/2.6.0/bin:$PATH
#ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.6.0
#ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.6.0
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

#COPY Gemfile* /fluentd/
#RUN buildDeps="sudo make gcc g++ libc-dev libffi-dev" \
#  runtimeDeps="" \
#      && apt-get update \
#     && apt-get upgrade -y \
#     && apt-get install \
#     -y --no-install-recommends \
#     $buildDeps $runtimeDeps net-tools \
#    && gem install bundler --version 2.1.4 \
#    && bundle config silence_root_warning true \
#    && bundle install --gemfile=/fluentd/Gemfile --path=/fluentd/vendor/bundle \
#    && SUDO_FORCE_REMOVE=yes \
#    apt-get purge -y --auto-remove \
#                  -o APT::AutoRemove::RecommendsImportant=false \
#                  $buildDeps \
# && rm -rf /var/lib/apt/lists/* \
#    && gem sources --clear-all \
#    && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install elasticsearch -v 8.6 \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem install fluent-plugin-multi-format-parser \
 && sudo gem install fluent-plugin-concat \
 && sudo gem install fluent-plugin-grok-parser \
 && sudo gem install fluent-plugin-prometheus \
 && sudo gem install fluent-plugin-json-in-json-2 \
 && sudo gem install fluent-plugin-record-modifier \
 && sudo gem install fluent-plugin-detect-exceptions \
 && sudo gem install fluent-plugin-rewrite-tag-filter \
 && sudo gem install fluent-plugin-dedot_filter \
 && sudo gem install fluent-plugin-kubernetes_metadata_filter \
 && sudo gem install ffi \
 && sudo gem install fluent-plugin-systemd \
 && sudo gem install fluent-plugin-parser-cri \
 && sudo gem install fluent-plugin-azure-storage-append-blob \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

ADD conf /fluentd/etc/
ADD plugins /fluentd/plugins/

COPY entrypoint.sh /fluentd/entrypoint.sh

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

EXPOSE 9292:9292

# Overwrite ENTRYPOINT to run fluentd as root for /var/log / /var/lib
ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]

I did also try using FROM fluent/fluentd:v1.16-debian-1 but there was no change there. I saw that there was a PR 3 days ago related to unpinning the version of faraday, perhaps that is related to this?

github-actions[bot] commented 1 year ago

@danfinn this issue was automatically closed because it did not follow the issue template.