search

kube-logging / logging-operator

Logging operator for Kubernetes
https://kube-logging.dev
Apache License 2.0
1.53k stars 327 forks source link

Gem::ConflictError when applying ClusterFlow #1251

Closed sebastiangaiser closed 1 year ago

sebastiangaiser commented 1 year ago

Describe the bug: FluentD leads to crashloopbackoff and FluentD-configcheck fails

Expected behaviour: FluentD is not crashing :D

Steps to reproduce the bug:

  1. Install operator via Helm with clusteroutput (in my case opensearch)
  2. Apply ClusterFlow leads to errors as shown

Additional context:

Environment details:

FluentD spec (also v1.15 tested) taken from Logging:

disablePvc: true
image: 
  tag: v1.14

/kind bug

ahma commented 1 year ago

@sebastiangaiser, thanks for the report. Could you recheck it with ghcr.io/kube-logging/fluentd:v1.14.6 ?

sebastiangaiser commented 1 year ago 
kubectl describe po -n logging logging-fluentd-configcheck-d77653a3 | grep Image
    Image:         ghcr.io/kube-logging/fluentd:v1.14.6
2023-03-17 10:47:48 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @raw
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-03-17 10:47:48 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-03-17 10:47:48 +0000 [info]: starting fluentd-1.15.3 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

Same error for the FluentD pod(s)...

ahma commented 1 year ago

@sebastiangaiser Thanks. Please check with v1.14.6-build.51 and send the logging output definition as well. (without the sensitive data. ;) )

sebastiangaiser commented 1 year ago

@ahma 

Failed to pull image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": failed to resolve reference "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": ghcr.io/banzaicloud/fluentd:v1.14.6-build.51: not found
pepov commented 1 year ago

@sebastiangaiser it's ghcr.io/kube-logging/fluentd:v1.14.6-build.51

sebastiangaiser commented 1 year ago

Sorry for the late reply:

kubectl describe po -n logging logging-fluentd-configcheck-29ea88f7 | grep Image  
    Image:         ghcr.io/kube-logging/fluentd:v1.14.6-build.51

Same error:

fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-04-11 13:29:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-04-11 13:29:04 +0000 [info]: starting fluentd-1.14.6 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
        from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

The ClusterOutput:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: general
  namespace: logging
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      valueFrom:
        secretKeyRef:
          key: password
          name: credentials
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user
pepov commented 1 year ago

I just ran this both with the latest ghcr.io/kube-logging/fluentd:v1.14 and ghcr.io/kube-logging/fluentd:v1.15 images.

My manifest:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: opensearch
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: default
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: opensearch
spec:
  match:
    - select: {}
  localOutputRefs:
    - "opensearch"
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: opensearch
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      value: asd
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user

The container is running although it is unable to connect to the host, so I guess it passes the error @sebastiangaiser mentioned:

2023-04-12 13:59:47 +0000 [info]: starting fluentd-1.14.6 pid=7 ruby="2.7.8"
...
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).
2023-04-12 14:13:08 +0000 [info]: starting fluentd-1.15.3 pid=7 ruby="2.7.8"
...
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 12. Retry to communicate after 8 second(s).
sebastiangaiser commented 1 year ago

My error occurs earlier than your Could not communicate to OpenSearch (in the configcheck).

The only difference I can see, you are using normal Flow and Output resources. I did it with ClusterFlow and ClusterOutput but I will try to make a minimal example as you did and paste the result here because I was using https://github.com/kube-logging/helm-charts/tree/main/charts/logging-operator-logging for deploying the CRs.

sebastiangaiser commented 1 year ago

Same error with these manifests and operator 4.X (helm upgrade --install -n logging logging-operator kube-logging/logging-operator --version 4.1.0-dev.2):

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
  namespace: logging
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: logging
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: opensearch
  namespace: logging
spec:
  match:
    - select: {}
  globalOutputRefs:
    - opensearch
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: opensearch
  namespace: logging
spec:
  opensearch:
    buffer:
      flush_interval: 10s
      flush_mode: interval
    hosts: hosts
    include_timestamp: true
    log_os_400_reason: true
    logstash_dateformat: '%Y%m%d'
    logstash_format: true
    logstash_prefix: some-prefix
    password:
      value: asd
    port: 443
    reload_connections: false
    request_timeout: 20s
    scheme: https
    suppress_type_name: true
    user: user
sebastiangaiser commented 1 year ago

@pepov it seams to work now with logging-operator 4.1.0 and logging-operator-logging 4.1.0 and ghcr.io/kube-logging/fluentd:v1.14.

Have you changed something to the images? In case, is there a best practice for pinning the image version for the logging-operator. I remember such a image version pinning for rook-ceph with the recommendation to pin it to a build date (for production environments).

pepov commented 1 year ago

We don't change the operator image version, so you don't have to worry about that 4.1.0 will stay the same. We have a different build and release setup for the kube-logging/fluentd:v1.14 and kube-logging/fluentd:v1.15 images, but that is going to change once kube-logging/fluentd:v1.15-staging get's merged into v1.15 and all the gem versions will be locked.

And to answer you question I beleive the updates in the elasticsearch/opensearch plugins fixed the conflict issue:

diff -dy --suppress-common-lines <( docker run --rm ghcr.io/kube-logging/fluentd:v1.14-build.51 gem list ) <( docker run --rm ghcr.io/kube-logging/fluentd:v1.15 gem list )
addressable (2.8.1)                     |       addressable (2.8.4)
aws-partitions (1.728.0)                    |       aws-partitions (1.745.0)
aws-sdk-core (3.170.0)                      |       aws-sdk-core (3.171.0)
aws-sdk-s3 (1.119.1)                        |       aws-sdk-s3 (1.120.1)
bundler (2.4.8)                         |       bundler (2.4.12)
connection_pool (2.3.0)                     |       connection_pool (2.4.0)
elasticsearch (7.13.3)                      |       elastic-transport (8.2.1)
elasticsearch-api (7.13.3)                  |       elasticsearch (8.7.0, 7.13.3)
                                >       elasticsearch-api (8.7.0, 7.13.3)
faraday-excon (1.1.0)                       |       faraday-excon (2.1.0, 1.1.0)
faraday_middleware-aws-sigv4 (0.6.1, 0.3.0)         |       faraday_middleware-aws-sigv4 (1.0.1, 0.3.0)
fluent-plugin-elasticsearch (5.2.4)             |       fluent-plugin-elasticsearch (5.3.0)
                                >       fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-opensearch (1.0.10)               |       fluent-plugin-opensearch (1.1.0)
fluent-plugin-splunk-hec (1.3.1)                |       fluent-plugin-splunk-hec (1.3.2)
                                >       fluent-plugin-vmware-log-intelligence (2.0.6)
                                >       fluent-plugin-vmware-loginsight (1.4.0)
fluentd (1.14.6)                        |       fluentd (1.15.3)
googleauth (1.3.0)                      |       googleauth (1.5.1)
msgpack (1.6.1)                         |       msgpack (1.7.0)
net-http-persistent (4.0.1, 3.1.0)              |       myslog (0.1.1)
                                >       net-http-persistent (4.0.2, 3.1.0)
nio4r (2.5.8)                           |       nio4r (2.5.9)
nokogiri (1.14.2)                       |       nokogiri (1.14.3)
opensearch-api (2.1.0)                      |       opensearch-api (2.2.0)
                                >       snappy (0.2.0)
tzinfo (2.0.6, 1.2.7)                       |       tzinfo (2.0.6, 1.2.11)
tzinfo-data (1.2022.7)                      |       tzinfo-data (1.2023.3)
uri (default: 0.10.0)                       |       uri (default: 0.10.0.2)

https://github.com/uken/fluent-plugin-elasticsearch/commit/1e817c9099459c6bae619a392955df54ce70ed3b https://github.com/fluent/fluent-plugin-opensearch/commit/9774a1cad2fdd32c74882351fcb4d2f4b1e26b13

sebastiangaiser commented 1 year ago

Thank you for the support. I found another issue that might be related: https://github.com/uken/fluent-plugin-elasticsearch/issues/1014

The question about the tagging was more in the direction for the fluentd images like ghcr.io/kube-logging/fluentd:v1.14. Is the tag v1.14 immutable (I assume not)? How about v1.14.6? Or would it be better to use the build number like v1.14.6-build.67?