search

fluent / fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Apache License 2.0
579 stars 246 forks source link

bug: fluentd won't start with error: /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:340:in `find_label': @a2170d34e9940ec56d328100e375c43e label not found (ArgumentError) #995

Open zteddie opened 10 months ago

zteddie commented 10 months ago

Describe the issue

I am trying to deploy fluent-operator (fluentd only) but it is failing. I tried both v1.15.3 and v1.14.6. I hope you can give me some help on it and see what I did wrong … here is how I deployed it:

helm install fluent-operator --create-namespace -n fluent fluent/fluent-operator  --set containerRuntime=containerd --set fluentd.enable=true --set fluentbit.enabled=false --set fluentd.image.repository=kubesphere/fluentd --set fluentd.image.tag="v1.14.6" ---> also tried v1.15.3:

Here is the error:

level=info msg="Fluentd started"
2023-11-16 20:33:18 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-11-16 20:33:18 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-11-16 20:33:18 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.10'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-11-16 20:33:18 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:340:in `find_label': @a2170d34e9940ec56d328100e375c43e label not found (ArgumentError)
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_emitter.rb:56:in `event_emitter_router'
    from /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-label-router-0.2.10/lib/fluent/plugin/out_label_router.rb:210:in `block in configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-label-router-0.2.10/lib/fluent/plugin/out_label_router.rb:209:in `each'
    from /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-label-router-0.2.10/lib/fluent/plugin/out_label_router.rb:209:in `configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:187:in `configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:132:in `add_match'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in `block in configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `each'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in `configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in `configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in `configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in `run_configure'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in `run_supervisor'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in `<top (required)>'
    from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
    from /usr/bin/fluentd:25:in `load'
    from /usr/bin/fluentd:25:in `<main>'
level=error msg="Fluentd exited" error="exit status 1"
level=info msg=backoff delay=1s

To Reproduce

see above

Expected behavior

fluentd started successfully

Your Environment

- Fluent Operator version:

cat Chart.yaml
apiVersion: v2
appVersion: 2.5.0
description: Fluent Operator provides great flexibility in building a logging layer
  based on Fluent Bit and Fluentd.
home: https://www.fluentd.org/
icon: https://raw.githubusercontent.com/fluent/fluent-operator/master/docs/images/fluent-operator-icon.svg
keywords:
- logging
- fluent-bit
- fluentd
- operator
maintainers:
- email: dehaocheng@kubesphere.io
  name: wenchajun
name: fluent-operator
sources:
- https://github.com/fluent/fluent-operator
version: 2.5.0

How did you install fluent operator?

helm install

Additional context

No response

jgrossmac commented 7 months ago

For anyone else that came across this issue like I did, it appears this error is the result of not having any active/correctly configured outputs (namespaced or cluster-wide). Once I created those, this error went away in the fluentd pods.