search

kube-logging / logging-operator

Logging operator for Kubernetes
https://kube-logging.dev
Apache License 2.0
1.53k stars 327 forks source link

Gem conflict in fluentd when using both elasticsearch and splunk_hec fluentd store #907

Closed BadLiveware closed 2 years ago

BadLiveware commented 2 years ago

Describe the bug: The fluend-configcheck pod is unable to check the configuration as there appears to be a gem conflict with net-http-persistent

Expected behaviour: fluentd to check the config

Steps to reproduce the bug: Try to use both elasticsearch store and splunk_hec at the same time

Additional context: The conflict seems to be between net-http-persistent-4.0.1 and ~>3.1 (used by fluent-plugin-splunk-hec-1.2.8) reproduced locally with below minimal config with ghcr.io/banzaicloud/fluentd:v1.13.3-alpine-14

Environment details:

Minimal configuration to reproduce:

# /fluentd/etc/fluent.conf

# include other config files
@include /fluentd/etc/generated.conf
# /fluentd/etc/generated.conf

<source>
  @type forward
  @id main_forward
  bind 0.0.0.0
  port 24240
</source>
<label @8f83eb89f0d625d474c102350356d1a3>
  <match **>
    @type copy
    <store>
      @type elasticsearch
    </store>
    <store>
      @type splunk_hec
    </store>
  </match>
</label>

Logs:

/fluentd/etc $ fluentd -c /fluentd/etc/fluent.conf --dry-run
2022-01-14 09:16:13 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.2'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.4'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.16'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-kafka' version '0.17.3'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.8'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.3'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2022-01-14 09:16:13 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2022-01-14 09:16:13 +0000 [info]: gem 'fluentd' version '1.13.3'
2022-01-14 09:16:13 +0000 [info]: gem 'fluentd' version '1.12.2'
2022-01-14 09:16:13 +0000 [info]: starting fluentd-1.13.3 as dry run mode ruby="2.7.5"
Traceback (most recent call last):
        32: from /usr/bin/fluentd:23:in `<main>'
        31: from /usr/bin/fluentd:23:in `load'
        30: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
        29: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        28: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        27: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
        26: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
        25: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
        24: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
        23: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
        22: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
        21: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
        20: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
        19: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
        18: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
        17: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
        16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in `add_match'
        15: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
        14: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
        13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `configure'
        12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `each'
        11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:72:in `block in configure'
        10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
         9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
         8: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
         7: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
         6: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
         5: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
         4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         2: from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.8/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
         1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
        35: from /usr/bin/fluentd:23:in `<main>'
        34: from /usr/bin/fluentd:23:in `load'
        33: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
        32: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        31: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        30: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
        29: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
        28: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
        27: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
        26: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
        25: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
        24: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
        23: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
        22: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
        21: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
        20: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
        19: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in `add_match'
        18: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
        17: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
        16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `configure'
        15: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `each'
        14: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:72:in `block in configure'
        13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
        12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
        11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
        10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
         9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
         8: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
         7: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         6: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         5: from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.8/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
         4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
         3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
         2: from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
         1: from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-splunk-hec-1.2.8, because net-http-persistent-4.0.1 conflicts with net-http-persistent (~> 3.1) (Gem::ConflictError)
        36: from /usr/bin/fluentd:23:in `<main>'
        35: from /usr/bin/fluentd:23:in `load'
        34: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
        33: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        32: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        31: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
        30: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
        29: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
        28: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
        27: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
        26: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
        25: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
        24: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
        23: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
        22: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
        21: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
        20: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in `add_match'
        19: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
        18: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
        17: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `configure'
        16: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `each'
        15: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:72:in `block in configure'
        14: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
        13: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
        12: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
        11: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
        10: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
         9: from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
         8: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         7: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         6: from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.8/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
         5: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
         4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
         3: from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
         2: from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
         1: from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-splunk-hec-1.2.8, because net-http-persistent-4.0.1 conflicts with net-http-persistent (~> 3.1) (Gem::ConflictError)

Gem (reverse) dependencies for net-http-persistent:

/ $ gem dependency net-http-persistent --reverse-dependencies
Gem net-http-persistent-3.1.0
  connection_pool (~> 2.2)
  hoe (~> 3.17, development)
  hoe-bundler (~> 1.5, development)
  hoe-travis (~> 1.4, >= 1.4.1, development)
  minitest (~> 5.11, development)
  rdoc (>= 4.0, < 7, development)
  Used by
    elasticsearch-transport-7.13.3 (net-http-persistent (>= 0, development))
    faraday-net_http_persistent-1.2.0 (net-http-persistent (>= 3.1, development))
    fluent-plugin-datadog-0.14.0 (net-http-persistent (~> 3.1))
    fluent-plugin-logzio-0.0.21 (net-http-persistent (~> 3.1))
    fluent-plugin-splunk-hec-1.2.8 (net-http-persistent (~> 3.1))

Gem net-http-persistent-4.0.1
  connection_pool (~> 2.2)
  hoe (~> 3.22, development)
  hoe-bundler (~> 1.5, development)
  hoe-travis (~> 1.4, >= 1.4.1, development)
  minitest (~> 5.14, development)
  rdoc (>= 4.0, < 7, development)
  Used by
    azure-storage-common-2.0.4 (net-http-persistent (~> 4.0))
    elasticsearch-transport-7.13.3 (net-http-persistent (>= 0, development))
    faraday-net_http_persistent-1.2.0 (net-http-persistent (>= 3.1, development))
    fluent-plugin-enhance-k8s-metadata-2.0.0 (net-http-persistent (~> 4.0))
    fluent-plugin-kubernetes-metadata-filter-2.5.3 (net-http-persistent (~> 4.0))

/kind bug

semos7 commented 2 years ago

Hey! Also just letting you know that we've also run into this.

Cheers

2022-03-07 10:01:13 +0000 [info]: starting fluentd-1.13.3 as dry run mode ruby="2.7.4"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-splunk-hec-1.2.7, because net-http-persistent-4.0.1 conflicts with net-http-persistent (~> 3.1) (Gem::ConflictError)
    from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
    from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
    from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.7/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
    from /usr/bin/fluentd:23:in `load'
    from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-splunk-hec-1.2.7, because net-http-persistent-4.0.1 conflicts with net-http-persistent (~> 3.1) (Gem::ConflictError)
    from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
    from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.7/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
    from /usr/bin/fluentd:23:in `load'
    from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-splunk-hec-1.2.7/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
    from /usr/bin/fluentd:23:in `load'
    from /usr/bin/fluentd:23:in `<main>'
semos7 commented 2 years ago

I think it comes down to what the fluent.conf loads first. Note, have not been able to really try it out properly, but it sure looks like it.

I was puzzled as our staging environment have the same version and same blend of elastic and splunk, but there it works. Same dependency trees and all and load order of the gems when fluentd starts up.

But when I compared the /fluentd/etc/generated.conf for both stage and prod, in stage the first plugin to be called is splunk_hec So a label like this is the first label to appear in the generated conf. 

<label @b9120fac26eee5d4c4c1b89613df1b49>
  <filter **>
...
  </filter>
  <match kubernetes.**>
...
    format ${namespace_name}.${pod_name}.${container_name}
  </match>
  <match **>
    @type splunk_hec
...
  </match>
</label>

While in Prod, the first label in the generated conf file is 

<label @92930c1e8f60cc3e165dd2b94d6ea0cb>
  <match kubernetes.**>
...
  </match>
  <match **>
    @type elasticsearch
...
  </match>
</label>
ronhorton commented 2 years ago

experiencing the same issue here at rancher. when elasticsearch and splunk are used as separate outputs, there's an error in the configcheck pod /usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in raise_if_conflicts': Unable to activate fluent-plugin-splunk-hec-1.2.10, because net-http-persistent-4.0.1 conflicts with net-http-persistent (~> 3.1) (Gem::ConflictError)