search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.91k stars 1.34k forks source link

Compatibility issues with fluentd plugins #4645

Open PranavBhatSF opened 1 month ago

PranavBhatSF commented 1 month ago

Usecase

We need to push kubernetes pod logs to Azure, S3 or Splunk based on the configuration.

What we have done

Built a custom image with Fluentd image as base and included the required plugins. Dockerfile

FROM ghcr.io/kube-logging/fluentd:v1.16-4.9-full as base-fluentd

# add azure storage append blob plugin
RUN gem install fluent-plugin-azure-storage-append-blob -v 0.2.1
# add aws s3 plugin
RUN gem install fluent-plugin-s3 -v 1.7.2
# add splunk plugin
RUN gem install fluent-plugin-splunk-hec -v 1.3.2

Tried with https://github.com/fluent/fluent-plugin-splunk as well. But hitting the same issue.

Issue

Fluentd is pushing logs to S3 and Azure. But when we configure splunk as destination, fluentd-configcheck pod fails with the below error

....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)

To Reproduce

Expected behavior

Fluentd should send logs to Splunk server

Your Environment

- Fluentd version: 1.16.3

This is a kubernetes 1.30.1 environment. But this is reproducible in older kubernetes versions as well.
We are using kube-logging stack. kube-logging version is 4.9.1

Your Configuration

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: splunk-output
spec:
  splunkHec:
    buffer:
      tags: '[]'
      timekey: <splunk_hec_timekey>
      timekey_use_utc: true
      timekey_wait: 10s
      type: file
    hec_host: <splunk_hec_host>
    hec_port: <splunk_hec_port>
    hec_token: <splunk_hec_token>
    index: <splunk_hec_index>
    insecure_ssl: true
    protocol: <splunk_hec_protocol>
    source: <splunk_hec_source>
    sourcetype: <splunk_hec_source_type>
EOF
kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: splunk-flow
  namespace: logging
spec:
  filters:
  - tag_normaliser:
      format: ${namespace_name}/${pod_name}.${container_name}
  globalOutputRefs:
  - splunk-output
  match:
  - select:
      container_names:
      - istio-proxy
      namespaces:
      - istio-system
  - exclude:
      container_names:
      - istio-proxy
      - istio-init
      - aicenter-hit-count-update
      - istio-configure-executor
      - on-prem-tenant-license-update
      - curl
      - recovery
      - aicenter-oob-scheduler
      - cert-trustor
  - exclude:
      namespaces:
      - default
  - exclude:
      labels:
        app: csi-snapshotter
  - exclude:
      labels:
        app: csi-resizer
  - select: {}
EOF

Your Error Log

➜  ~ kubectl logs logging-fluentd-configcheck-691ab8a2  -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
2024-09-25 07:20:53 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-25 07:20:53 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gelf-best' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.9.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.6.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.2'
2024-09-25 07:20:53 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.2.4"
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems.rb:210:in `rescue in try_activate'
    from /usr/local/lib/ruby/3.2.0/rubygems.rb:203:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
    from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
    from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
    from /usr/local/lib/ruby/3.2.0/rubygems.rb:204:in `try_activate'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
    from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
    from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
    from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
    from /usr/local/bundle/bin/fluentd:25:in `load'
    from /usr/local/bundle/bin/fluentd:25:in `<main>'
daipom commented 1 month ago

Thanks for your report! It appears that some dependencies conflict with each other. We need to check it.

daipom commented 1 month ago

At least, fluent-plugin-azure-storage-append-blob requires faraday v1. On the other hand, swd v2 requires faraday v2. Using swd v1 may solve the error.