Closed sebastiangaiser closed 1 year ago
@sebastiangaiser, thanks for the report. Could you recheck it with ghcr.io/kube-logging/fluentd:v1.14.6 ?
kubectl describe po -n logging logging-fluentd-configcheck-d77653a3 | grep Image
Image: ghcr.io/kube-logging/fluentd:v1.14.6
2023-03-17 10:47:48 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @raw
2023-03-17 10:47:48 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-03-17 10:47:48 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-03-17 10:47:48 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-17 10:47:48 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-03-17 10:47:48 +0000 [info]: starting fluentd-1.15.3 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
Same error for the FluentD pod(s)...
@sebastiangaiser Thanks. Please check with v1.14.6-build.51 and send the logging output definition as well. (without the sensitive data. ;) )
@ahma
Failed to pull image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": failed to resolve reference "ghcr.io/banzaicloud/fluentd:v1.14.6-build.51": ghcr.io/banzaicloud/fluentd:v1.14.6-build.51: not found
@sebastiangaiser it's ghcr.io/kube-logging/fluentd:v1.14.6-build.51
Sorry for the late reply:
kubectl describe po -n logging logging-fluentd-configcheck-29ea88f7 | grep Image
Image: ghcr.io/kube-logging/fluentd:v1.14.6-build.51
Same error:
fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-04-11 13:29:04 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.1'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-04-11 13:29:04 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-04-11 13:29:04 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-04-11 13:29:04 +0000 [info]: starting fluentd-1.14.6 as dry run mode ruby="2.7.7"
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.7.4 conflicts with faraday (~> 1.10) (Gem::ConflictError)
from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
from /usr/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- fluent/log-ext (LoadError)
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.10/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:23:in `load'
from /usr/bin/fluentd:23:in `<main>'
The ClusterOutput:
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: general
namespace: logging
spec:
opensearch:
buffer:
flush_interval: 10s
flush_mode: interval
hosts: hosts
include_timestamp: true
log_os_400_reason: true
logstash_dateformat: '%Y%m%d'
logstash_format: true
logstash_prefix: some-prefix
password:
valueFrom:
secretKeyRef:
key: password
name: credentials
port: 443
reload_connections: false
request_timeout: 20s
scheme: https
suppress_type_name: true
user: user
I just ran this both with the latest ghcr.io/kube-logging/fluentd:v1.14
and ghcr.io/kube-logging/fluentd:v1.15
images.
My manifest:
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
name: opensearch
spec:
fluentd: {}
fluentbit: {}
controlNamespace: default
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: opensearch
spec:
match:
- select: {}
localOutputRefs:
- "opensearch"
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: opensearch
spec:
opensearch:
buffer:
flush_interval: 10s
flush_mode: interval
hosts: hosts
include_timestamp: true
log_os_400_reason: true
logstash_dateformat: '%Y%m%d'
logstash_format: true
logstash_prefix: some-prefix
password:
value: asd
port: 443
reload_connections: false
request_timeout: 20s
scheme: https
suppress_type_name: true
user: user
The container is running although it is unable to connect to the host, so I guess it passes the error @sebastiangaiser mentioned:
2023-04-12 13:59:47 +0000 [info]: starting fluentd-1.14.6 pid=7 ruby="2.7.8"
...
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:21 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:06:25 +0000 [warn]: #0 [logging::opensearch:clusteroutput:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).
2023-04-12 14:13:08 +0000 [info]: starting fluentd-1.15.3 pid=7 ruby="2.7.8"
...
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:11 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 14. Retry to communicate after 2 second(s).
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:15 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 13. Retry to communicate after 4 second(s).
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Could not communicate to OpenSearch, resetting connection and trying again. no address for hosts (Resolv::ResolvError)
2023-04-12 14:13:23 +0000 [warn]: #0 [flow:default:opensearch:output:default:opensearch] Remaining retry: 12. Retry to communicate after 8 second(s).
My error occurs earlier than your Could not communicate to OpenSearch
(in the configcheck).
The only difference I can see, you are using normal Flow and Output resources. I did it with ClusterFlow and ClusterOutput but I will try to make a minimal example as you did and paste the result here because I was using https://github.com/kube-logging/helm-charts/tree/main/charts/logging-operator-logging for deploying the CRs.
Same error with these manifests and operator 4.X (helm upgrade --install -n logging logging-operator kube-logging/logging-operator --version 4.1.0-dev.2
):
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
name: logging
namespace: logging
spec:
fluentd: {}
fluentbit: {}
controlNamespace: logging
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
name: opensearch
namespace: logging
spec:
match:
- select: {}
globalOutputRefs:
- opensearch
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: opensearch
namespace: logging
spec:
opensearch:
buffer:
flush_interval: 10s
flush_mode: interval
hosts: hosts
include_timestamp: true
log_os_400_reason: true
logstash_dateformat: '%Y%m%d'
logstash_format: true
logstash_prefix: some-prefix
password:
value: asd
port: 443
reload_connections: false
request_timeout: 20s
scheme: https
suppress_type_name: true
user: user
@pepov it seams to work now with logging-operator 4.1.0
and logging-operator-logging 4.1.0
and ghcr.io/kube-logging/fluentd:v1.14
.
Have you changed something to the images? In case, is there a best practice for pinning the image version for the logging-operator. I remember such a image version pinning for rook-ceph with the recommendation to pin it to a build date (for production environments).
We don't change the operator image version, so you don't have to worry about that 4.1.0 will stay the same. We have a different build and release setup for the kube-logging/fluentd:v1.14
and kube-logging/fluentd:v1.15
images, but that is going to change once kube-logging/fluentd:v1.15-staging
get's merged into v1.15
and all the gem versions will be locked.
And to answer you question I beleive the updates in the elasticsearch/opensearch plugins fixed the conflict issue:
diff -dy --suppress-common-lines <( docker run --rm ghcr.io/kube-logging/fluentd:v1.14-build.51 gem list ) <( docker run --rm ghcr.io/kube-logging/fluentd:v1.15 gem list )
addressable (2.8.1) | addressable (2.8.4)
aws-partitions (1.728.0) | aws-partitions (1.745.0)
aws-sdk-core (3.170.0) | aws-sdk-core (3.171.0)
aws-sdk-s3 (1.119.1) | aws-sdk-s3 (1.120.1)
bundler (2.4.8) | bundler (2.4.12)
connection_pool (2.3.0) | connection_pool (2.4.0)
elasticsearch (7.13.3) | elastic-transport (8.2.1)
elasticsearch-api (7.13.3) | elasticsearch (8.7.0, 7.13.3)
> elasticsearch-api (8.7.0, 7.13.3)
faraday-excon (1.1.0) | faraday-excon (2.1.0, 1.1.0)
faraday_middleware-aws-sigv4 (0.6.1, 0.3.0) | faraday_middleware-aws-sigv4 (1.0.1, 0.3.0)
fluent-plugin-elasticsearch (5.2.4) | fluent-plugin-elasticsearch (5.3.0)
> fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-opensearch (1.0.10) | fluent-plugin-opensearch (1.1.0)
fluent-plugin-splunk-hec (1.3.1) | fluent-plugin-splunk-hec (1.3.2)
> fluent-plugin-vmware-log-intelligence (2.0.6)
> fluent-plugin-vmware-loginsight (1.4.0)
fluentd (1.14.6) | fluentd (1.15.3)
googleauth (1.3.0) | googleauth (1.5.1)
msgpack (1.6.1) | msgpack (1.7.0)
net-http-persistent (4.0.1, 3.1.0) | myslog (0.1.1)
> net-http-persistent (4.0.2, 3.1.0)
nio4r (2.5.8) | nio4r (2.5.9)
nokogiri (1.14.2) | nokogiri (1.14.3)
opensearch-api (2.1.0) | opensearch-api (2.2.0)
> snappy (0.2.0)
tzinfo (2.0.6, 1.2.7) | tzinfo (2.0.6, 1.2.11)
tzinfo-data (1.2022.7) | tzinfo-data (1.2023.3)
uri (default: 0.10.0) | uri (default: 0.10.0.2)
https://github.com/uken/fluent-plugin-elasticsearch/commit/1e817c9099459c6bae619a392955df54ce70ed3b https://github.com/fluent/fluent-plugin-opensearch/commit/9774a1cad2fdd32c74882351fcb4d2f4b1e26b13
Thank you for the support. I found another issue that might be related: https://github.com/uken/fluent-plugin-elasticsearch/issues/1014
The question about the tagging was more in the direction for the fluentd images like ghcr.io/kube-logging/fluentd:v1.14
. Is the tag v1.14
immutable (I assume not)? How about v1.14.6
? Or would it be better to use the build number like v1.14.6-build.67
?
Describe the bug: FluentD leads to crashloopbackoff and FluentD-configcheck fails
Expected behaviour: FluentD is not crashing :D
Steps to reproduce the bug:
Additional context:
Environment details:
Logs from the misbehaving component (and any other relevant logs): Output of
kubectl logs -f -n logging logging-operator-logging-fluentd-configcheck-dcbaaaf3
FluentD spec (also
v1.15
tested) taken from Logging:/kind bug