search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 217 forks source link

gem dependency conflict #415

Open fabifont opened 1 year ago

fabifont commented 1 year ago

Describe the bug

I am running RUN fluent-gem install elasticsearch -v 8.6.0 && fluent-gem install fluent-plugin-elasticsearch fluent-plugin-s3 elasticsearch-api in my Dockerfile, two versions of faraday will be installed (2.7.3, 1.10.3) and this will generate a conflict when fluentd starts.

To Reproduce

Dockerfile:

FROM fluentd:v1.14-1

USER root
RUN fluent-gem install elasticsearch -v 8.6.0 \
    && fluent-gem install fluent-plugin-elasticsearch fluent-plugin-s3 elasticsearch-api

docker-compose.yml:

version: "3.9"

services:
  fluentd:
    image: fluentd
    build:
      context: fluentd
      dockerfile: Dockerfile
    container_name: fluentd
    environment:
      - AWS_KEY_ID=${S3_ACCESS_KEY}
      - AWS_SECRET_KEY=${S3_SECRET_KEY}
      - S3_BUCKET=${LOGS_BUCKET}
      - S3_REGION=${S3_REGION}
      - UPLOAD_INTERVAL=24h
      - UPLOAD_TYPE=json
      - TZ=${TZ}
    volumes:
      - ./fluentd/fluentd.conf:/fluentd/etc/fluent.conf
    ports:
      - 24224:24224
      - 24224:24224/udp
    restart: unless-stopped

Expected behavior

Fluentd starts without conflicts.

Your Environment

- Fluentd version: 1.14-1
- fluent-plugin-s3 version: 1.7.2
- aws-sdk-s3 version: 1.118.0
- aws-sdk-sqs version: 1.53.0

Your Configuration

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match infrastructure.*.logs>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    logstash_prefix fluentd-infrastructure
    logstash_dateformat %Y%m%d
    log_es_400_reason true
    <buffer>
      timekey 10s
      timekey_use_utc true
      timekey_wait 10m
      flush_interval 10s
    </buffer>
  </store>
  <store>
    @type s3
    aws_key_id "#{ENV['AWS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_KEY']}"
    s3_bucket "#{ENV['S3_BUCKET']}"
    s3_region "#{ENV['S3_REGION']}"
    path logs/infrastructure/
    buffer_path /var/log/fluent/s3
    time_slice_format %Y%m%d
    time_slice_wait "#{ENV['UPLOAD_INTERVAL']}"
    utc
    store_as json
    buffer_chunk_limit 256m
  </store>
  <store>
    @type stdout
  </store>
</match>

Your Error Log

Dockerfile logs:

#5 [ 2/20] RUN fluent-gem install elasticsearch -v 8.6.0     && fluent-gem install fluent-plugin-elasticsearch fluent-plugin-s3 elasticsearch-api
#5 2.979 Successfully installed faraday-net_http-3.0.2
#5 2.979 Successfully installed ruby2_keywords-0.0.5
#5 2.979 Successfully installed faraday-2.7.3
#5 2.979 Successfully installed multi_json-1.15.0
#5 2.979 Successfully installed elastic-transport-8.1.0
#5 2.979 Successfully installed elasticsearch-api-8.6.0
#5 2.979 Successfully installed elasticsearch-8.6.0
#5 2.979 7 gems installed
#5 16.13 Successfully installed faraday-em_http-1.0.0
#5 16.13 Successfully installed faraday-em_synchrony-1.0.0
#5 16.13 Successfully installed faraday-excon-1.1.0
#5 16.13 Successfully installed faraday-httpclient-1.0.1
#5 16.13 Successfully installed multipart-post-2.2.3
#5 16.13 Successfully installed faraday-multipart-1.0.4
#5 16.13 Successfully installed faraday-net_http-1.0.1
#5 16.13 Successfully installed faraday-net_http_persistent-1.2.0
#5 16.13 Successfully installed faraday-patron-1.0.0
#5 16.13 Successfully installed faraday-rack-1.0.0
#5 16.13 Successfully installed faraday-retry-1.0.3
#5 16.13 Successfully installed faraday-1.10.3
#5 16.13 Successfully installed excon-0.97.1
#5 16.13 Successfully installed fluent-plugin-elasticsearch-5.2.4
#5 16.13 Successfully installed jmespath-1.6.2
#5 16.13 Successfully installed aws-partitions-1.695.0
#5 16.13 Successfully installed aws-eventstream-1.2.0
#5 16.13 Successfully installed aws-sigv4-1.5.2
#5 16.13 Successfully installed aws-sdk-core-3.169.0
#5 16.13 Successfully installed aws-sdk-kms-1.62.0
#5 16.13 Successfully installed aws-sdk-s3-1.118.0
#5 16.13 Successfully installed aws-sdk-sqs-1.53.0
#5 16.13 Successfully installed fluent-plugin-s3-1.7.2
#5 16.13 Successfully installed elasticsearch-api-8.6.0
#5 16.13 24 gems installed
#5 DONE 16.2s]

docker-compose logs fluentd:

fluentd  | 2023-01-19 19:08:40 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd  | 2023-01-19 19:08:40 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
fluentd  | 2023-01-19 19:08:40 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
fluentd  | 2023-01-19 19:08:40 +0000 [info]: gem 'fluentd' version '1.14.0'
fluentd  | /usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.3 conflicts with faraday (~> 1.10) (Gem::ConflictError)
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:102:in `block in search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:44:in `lookup'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:169:in `new_impl'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:114:in `new_output'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:665:in `run_supervisor'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
fluentd  |  from /usr/bin/fluentd:23:in `load'
fluentd  |  from /usr/bin/fluentd:23:in `<main>'
fluentd  | /usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.3 conflicts with faraday (~> 1.10) (Gem::ConflictError)
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:102:in `block in search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:44:in `lookup'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:169:in `new_impl'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:114:in `new_output'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:665:in `run_supervisor'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
fluentd  |  from /usr/bin/fluentd:23:in `load'
fluentd  |  from /usr/bin/fluentd:23:in `<main>'
fluentd  | /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:102:in `block in search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:99:in `search'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/registry.rb:44:in `lookup'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:169:in `new_impl'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:114:in `new_output'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:108:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:665:in `run_supervisor'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd  |  from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
fluentd  |  from /usr/bin/fluentd:23:in `load'
fluentd  |  from /usr/bin/fluentd:23:in `<main>'

Additional context

No response

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

daipom commented 1 year ago

I don't know well about faraday and elastic-search, but isn't this an issue of fluent-plugin-elasticsearch?