ctrader321 commented 1 year ago

Bugs should be filed for issues encountered whilst operating logging-operator. You should first attempt to resolve your issues through the community support channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator Please provide as much detail as possible.

Describe the bug: When applying a flow and output to a namespace, the configcheck pod errors out with little information. Flow and Output objects have no reported 'Problems'. Logging-Operator logs indicate only that "the current config is invalid."

{"level":"error","ts":1680045283.611491,"logger":"controller.logging","msg":"Reconciler error","reconciler group":"logging.banzaicloud.io","reconciler kind":"Logging","name":"logging-operator-logging","namespace":"","error":"current config is invalid","errorVerbose":"current config is invalid\ngithub.com/banzaicloud/logging-operator/pkg/resources/fluentd.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/fluentd/fluentd.go:138\ngithub.com/banzaicloud/logging-operator/controllers/logging.(*LoggingReconciler).Reconcile\n\t/workspace/controllers/logging/logging_controller.go:149\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227"}

# k get po -n logging
NAME                                                    READY   STATUS    RESTARTS   AGE
ctc8-logging-operator-867c74f6b5-7wqzh                  1/1     Running   0          105m
logging-operator-logging-fluentd-configcheck-313b1d93   0/1     Error     0          24m

# k get flow,output -n ofp-devops-pytest
NAME                                  ACTIVE   PROBLEMS
flow.logging.banzaicloud.io/s3-flow   true

NAME                                      ACTIVE   PROBLEMS
output.logging.banzaicloud.io/s3-output   true

Expected behaviour: More detailed reason on why config is invalid. Also, which config is invalid? The logging object config? The flow? The output? I do not know.

Steps to reproduce the bug: Install logging-operator via helm template command on chart version 4.1.0-dev.1 with default values.

---
# Source: logging-operator/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctc8-logging-operator
  namespace: logging
  labels:
    app.kubernetes.io/name: logging-operator
    helm.sh/chart: logging-operator-3.17.10
    app.kubernetes.io/instance: ctc8
    app.kubernetes.io/version: "3.17.10"
    app.kubernetes.io/managed-by: Helm
---
# Source: logging-operator/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: ctc8-logging-operator
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
 ....
---
# Source: logging-operator/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ctc8-logging-operator
  labels:
    app.kubernetes.io/name: logging-operator
    helm.sh/chart: logging-operator-3.17.10
    app.kubernetes.io/instance: ctc8
    app.kubernetes.io/version: "3.17.10"
    app.kubernetes.io/managed-by: Helm
subjects:
  - kind: ServiceAccount
    name: ctc8-logging-operator
    namespace: logging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ctc8-logging-operator
---
# Source: logging-operator/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ctc8-logging-operator
  namespace: logging
  labels:
    app.kubernetes.io/name: logging-operator
    helm.sh/chart: logging-operator-3.17.10
    app.kubernetes.io/instance: ctc8
    app.kubernetes.io/version: "3.17.10"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 8080
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: logging-operator
    app.kubernetes.io/instance: ctc8
---
# Source: logging-operator/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctc8-logging-operator
  namespace: logging
  labels:
    app.kubernetes.io/name: logging-operator
    helm.sh/chart: logging-operator-3.17.10
    app.kubernetes.io/instance: ctc8
    app.kubernetes.io/version: "3.17.10"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: logging-operator
      app.kubernetes.io/instance: ctc8
  template:
    metadata:
      labels:
        app.kubernetes.io/name: logging-operator
        app.kubernetes.io/instance: ctc8
    spec:
      containers:
        - name: logging-operator
          image: ".../banzaicloud/logging-operator:3.17.10"
          args:
            - -enable-leader-election=true
          imagePullPolicy: IfNotPresent
          resources:
            {}
          ports:
            - name: http
              containerPort: 8080
      serviceAccountName: ctc8-logging-operator

Apply logging-operator-logging logging object using helm template command on logging-operator-logging chart version version: 4.0.0.

---
# Source: logging-operator-logging/templates/logging.yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging-operator-logging
  labels:
    app.kubernetes.io/name: logging-operator-logging
    helm.sh/chart: logging-operator-logging-4.0.0
    app.kubernetes.io/instance: ctc-dev-logging
    app.kubernetes.io/version: "4.0.0"
    app.kubernetes.io/managed-by: Helm
spec:
  clusterDomain: cluster.local
  controlNamespace: logging
  fluentd:
    tls:
      enabled: true
      secretName: logging-operator-logging-fluentd-tls
      sharedKey: ""
  fluentbit:
    tls:
      enabled: true
      secretName: logging-operator-logging-fluentbit-tls
      sharedKey: ""
  nodeAgents: []

At this point, initial configcheck pod which spins up, completes with no errors:

# k get pod logging-operator-logging-fluentd-configcheck-8a4ed2f3 -n logging
NAME                                                    READY   STATUS      RESTARTS   AGE
logging-operator-logging-fluentd-configcheck-8a4ed2f3   0/1     Completed   0          65s

# k logs logging-operator-logging-fluentd-configcheck-8a4ed2f3 -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-03-28 23:34:45 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.18'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-kafka' version '0.17.5'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.5'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.13'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-03-28 23:34:45 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-03-28 23:34:45 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-03-28 23:34:45 +0000 [info]: starting fluentd-1.14.6 as dry run mode ruby="2.7.6"
2023-03-28 23:34:45 +0000 [warn]: [main_forward] For security reason, setting private_key_passphrase is recommended when cert_path is specified
2023-03-28 23:34:45 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-28 23:34:45 +0000 [info]: using configuration file: <ROOT>
  <system>
    rpc_endpoint "127.0.0.1:24444"
    log_level info
    workers 1
  </system>
  <source>
    @type forward
    @id main_forward
    bind "0.0.0.0"
    port 24240
    <transport tls>
      ca_path "/fluentd/tls/ca.crt"
      cert_path "/fluentd/tls/tls.crt"
      client_cert_auth true
      private_key_path "/fluentd/tls/tls.key"
      version TLSv1_2
    </transport>
    <security>
      self_hostname "fluentd"
      shared_key xxxxxx
    </security>
  </source>
  <match **>
    @type label_router
    @id main
    metrics false
  </match>
  <label @ERROR>
    <match **>
      @type null
      @id main-fluentd-error
    </match>
  </label>
  <match **>
    @type null
    @id main-no-output
  </match>
</ROOT>
2023-03-28 23:34:45 +0000 [info]: finished dry run mode

Apply a barebones namespace scoped flow and output with necessary secrets:

Flow

k apply -f s3-flow.yaml

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: s3-flow
  namespace: ofp-devops-pytest
spec:
  filters:
    - tag_normaliser:
        format: ${namespace_name}.${pod_name}.${container_name}
  localOutputRefs:
    - s3-output
  match:
    - select: {}

Output

k apply -f s3-output.yaml 

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: s3-output
  namespace: "ofp-devops-pytest"
spec:
  s3:
    aws_key_id:
      valueFrom:
        secretKeyRef:
          name: s3-secret
          key: awsAccessKeyId
    aws_sec_key:
      valueFrom:
        secretKeyRef:
          name: s3-secret
          key: awsSecretAccessKey
    s3_bucket: logging-s3
    s3_region: us-standard
    s3_endpoint:
    path: logs/%Y/%m/%d/
    buffer:
      timekey: 5m
      timekey_wait: 30s
      timekey_use_utc: true

Additional context: Add any other context about the problem here.

Environment details:

Kubernetes version: v1.25.6
logging-operator version: 3.17.10
Install method: helm to generate --> apply manifests

helm template --namespace logging ctc banzaicloud-stable/logging-operator > logging-operator.yaml

Logs from the misbehaving component (and any other relevant logs):

k logs logging-operator-logging-fluentd-configcheck-313b1d93 -n logging

fluentd -c /fluentd/etc/fluent.conf --dry-run /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:56:in skip': invalid byte sequence in UTF-8 (ArgumentError) from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:56:inskip' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:84:in spacing' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:61:inparse_element' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:169:in block in eval_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:163:ineach' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:163:in eval_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:146:inparse_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:105:in parse_element' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:44:inparse!' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:33:in parse' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config.rb:58:inparse' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config.rb:39:in build' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:618:ininitialize' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:354:in new' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:354:in<top (required)>' from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:inrequire' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in <top (required)>' from /usr/bin/fluentd:23:inload' from /usr/bin/fluentd:23:in `

'


- Resource definition (possibly in YAML format) that caused the issue, without sensitive data:  Resource defs above.

/kind bug

ctrader321 commented 1 year ago

As an aside, are we able to configure image proxy anywhere via the values? I noticed the daemonset that is applied only has image referenced like so:

Image:      fluent/fluent-bit:1.9.5

pepov commented 1 year ago

Hey @ctrader321 sorry, but we can't help with supporting old versions and I wasn't able to reproduce with logging operator 4.0.0.

As for your image proxy question: what kind of proxy are you referring to?

pepov commented 1 year ago

closing this for now, feel free to reopen if you still have questions or want to clarify

suinxinliulang commented 9 months ago

@pepov I have an same issue. I want to learn logging operator, so I write a demo, but when I apply flow and output config yaml, same error will shows.

helm version: v3.13.2 logging-operator version: 4.4.2(Image: ghcr.io/kube-logging/logging-operator:4.4.2) k3s version: v1.27.6+k3s1

logging yaml:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: demo-log-collector
  namespace: logging
spec:
  controlNamespace: logging
  watchNamespaces: ["dev"]
  fluentd:
    disablePvc: true

fluentbitAgent yaml:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
    name: demo-log-collector
    namespace: logging
spec: {}

flow and output yaml:

apiVersion: v1
kind: Secret
metadata:
  name: elastic-pwd
  namespace: dev
type: Opaque
data:
  es-password: xxxxxxxxxxxxxxxxxxxxxxx
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: es-output
  namespace: dev
spec:
  elasticsearch:
    host: xxxxxxx
    port: xxxxx
    scheme: https
    ssl_verify: false
    ssl_version: TLSv1_2
    user: elastic
    password: 
      valueFrom:
        secretKeyRef:
          name: "elastic-pwd"
          key: "es-password"
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: exchange
  namespace: dev
spec:
  filters:
    - stdout: {}
  localOutputRefs:
    - es-output
  match:
    - select:
        labels:
          appName: logdemo

demo application yaml, it is a springboot app and just print log schedulely:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: logdemo-deployment
  labels:
    deployName: logdemo
spec:
  replicas: 1
  selector:
    matchLabels:
      appName: logdemo
  template:
    metadata:
      labels:
        appName: logdemo
    spec:
      containers:
      - name: logdemo
        image: swr.cn-north-1.myhuaweicloud.com/suixinliulang_dev/log-demo:1.0

error stack:

fan@k3s-master:~$ kubectl get pod -n logging
NAME                                              READY   STATUS      RESTARTS         AGE
demo-log-collector-fluentd-configcheck-ac2d4553   0/1     Completed   0                3h10m
demo-log-collector-fluentbit-j9mck                1/1     Running     0                122m
logging-operator-5898b6769b-6lllg                 1/1     Running     13 (3h39m ago)   3d3h
demo-log-collector-fluentd-0                      2/2     Running     0                3h10m
demo-log-collector-fluentd-configcheck-8fba9222   0/1     Error       0                23m
fan@k3s-master:~$ kubectl logs demo-log-collector-fluentd-configcheck-8fba9222 -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:56:in `skip': invalid byte sequence in UTF-8 (ArgumentError)
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:56:in `skip'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:84:in `spacing'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:61:in `parse_element'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:169:in `block in eval_include'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:in `each'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:in `eval_include'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:146:in `parse_include'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:105:in `parse_element'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:44:in `parse!'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:33:in `parse'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config.rb:71:in `parse'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config.rb:52:in `build'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:679:in `initialize'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in `new'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in `<top (required)>'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
        from /usr/bin/fluentd:25:in `load'
        from /usr/bin/fluentd:25:in `<main>'

pepov commented 9 months ago

Can't really tell without being able to check the actual config as is. It can be something with the secret you are using. I would try to see the config without the secret whether it works or not.

suinxinliulang commented 9 months ago

Can't really tell without being able to check the actual config as is. It can be something with the secret you are using. I would try to see the config without the secret whether it works or not.

fluentd config:

fan@k3s-master:~$ kubectl get secret demo-log-collector-fluentd-app -n logging -o jsonpath="{.data['fluentd\.conf']}" | base64 --decode
<source>
  @type forward
  @id main_forward
  bind 0.0.0.0
  port 24240
</source>
<match **>
  @type label_router
  @id main
  metrics false
</match>
<label @ERROR>
  <match **>
    @type null
    @id main-fluentd-error
  </match>
</label>

fluentbit config:

fan@k3s-master:~$ kubectl get secret demo-log-collector-fluentbit -n logging -o jsonpath="{.data['fluent-bit\.conf']}" | base64 --decode

[SERVICE]
    Flush        1
    Grace        5
    Daemon       Off
    Log_Level    info
    Parsers_File /fluent-bit/etc/parsers.conf
    Coro_Stack_Size    24576
    storage.path  /buffers

[INPUT]
    Name         tail
    DB  /tail-db/tail-containers-state.db
    DB.locking  true
    Mem_Buf_Limit  5MB
    Parser  cri
    Path  /var/log/containers/*.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kubernetes.*
[FILTER]
    Name        kubernetes
    Buffer_Size  0
    K8S-Logging.Exclude  On
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Tag_Prefix  kubernetes.var.log.containers
    Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Token_TTL  600
    Kube_URL  https://kubernetes.default.svc:443
    Match  kubernetes.*
    Merge_Log  On
    Use_Kubelet  Off
[OUTPUT]
    Name          forward
    Match         *
    Host          demo-log-collector-fluentd.logging.svc.cluster.local.
    Port          24240
    Retry_Limit  False

kube-logging / logging-operator

logging-operator-logging fluentd configcheck errors #1280

k logs logging-operator-logging-fluentd-configcheck-313b1d93 -n logging