bug: default parsers are not deployed from fluent-operator

[SvenThies]

Describe the issue

When deploying fluent-bit with the fluent-operator (plain vanilla with default values) the default parser (cri, json, etc.) are not deployed. As the cri parser is the default parser in the input: tail component when selecting kubernetes: true and containerRuntime: containerd.

To Reproduce

Install the chart v2.9.0 with helm (v3.15.2):

helm install fluent-operator-test fluent/fluent-operator --version 2.9.0 -f values-test.yaml

using the following values-test.yaml:

# Default values for fluentbit-operator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Set this to containerd or crio if you want to collect CRI format logs
containerRuntime: containerd
#  If you want to deploy a default Fluent Bit pipeline (including Fluent Bit Input, Filter, and output) to collect Kubernetes logs, you'll need to set the Kubernetes parameter to true
# see https://github.com/fluent/fluent-operator/tree/master/manifests/logging-stack
Kubernetes: true

operator:
  # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
  # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
  initcontainer:
    repository: "docker"
    tag: "20.10"

    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  container:
    repository: "kubesphere/fluent-operator"
    tag: "v2.9.0"
  resources:
    limits:
      cpu: 1000m
      memory: 1000Mi
    requests:
      cpu: 100m
      memory: 20Mi
  disableComponentControllers: fluentd

fluentbit:
  # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
  crdsEnable: true
  enable: true
  serviceMonitor: false
  image:
    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
    tag: "v3.0.7"
  resources:
    limits:
      cpu: 1000m
      memory: 1000Mi
    requests:
      cpu: 10m
      memory: 25Mi

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*.log"
      skipLongLines: true
      readFromHead: false

  # NOTE: we will use our own, modified filters and disable it here
  filter:
    multiline:
      enable: false
    kubernetes:
      enable: true
      labels: false
      annotations: false
    containerd:
      enable: true

  output:
    es:
      enable: false

fluentd:
  # Installs a sub chart carrying the CRDs for the fluentd controller. The sub chart is enabled by default.
  crdsEnable: false
  enable: false

nameOverride: ""
fullnameOverride: fluent-operator
namespaceOverride: ""

This produces the following error in the fluent-bit container:

[2024/06/26 14:03:13] [error] [input:tail:tail.1] parser 'cri' is not registered

Resulting in not correctly parsed logs from kubernetes in the format of containerd log files.

Expected behavior

An additional template that deploys the default parsers using the clusterparser CRD. Like so e.g.:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterParser
metadata:
  name: containerd
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  regex:
    timeKey: time
    timeFormat: "%Y-%m-%dT%H:%M:%S.%L%z" 
    regex: '^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$'
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterParser
metadata:
  name: json
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  json:
    timeKey: time
    timeFormat: "%d/%b/%Y:%H:%M:%S %z" 
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterParser
metadata:
  name: cri
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  # http://rubular.com/r/tjUt3Awgg4
  regex:
    timeKey: time
    timeFormat: "%Y-%m-%dT%H:%M:%S.%L%z" 
    timeKeep: true
    regex: '^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$'

Your Environment

- Fluent Operator version: v2.9.0
- Container Runtime: containerd
- Operating system: Fedora CoreOS 38
- Kernel version: 6.4.7

How did you install fluent operator?

No response

Additional context

No response

[SvenThies]

Update: The error is not specific to the above config. This also happens when I clone the repo (master) and install the helm chart with

helm install -f charts/fluent-operator/values.yaml fluent-operator charts/fluent-operator

The fluent-bit pod shows:

[2024/06/30 21:24:58] [error] [input:tail:tail.1] parser 'docker' is not registered

[cw-Guo]

i checked the image 3.0.7-debug, i found that parsers.conf and parsers_multiline.conf are empty.

# part of the fluent-bit configuration file
fluent-bit.conf: |-
  [Service]
      Http_Server    true
      Parsers_File    /fluent-bit/config/parsers.conf
      Parsers_File    /fluent-bit/config/parsers_multiline.conf

# the parser files
root@fluent-bit-fnj2t:/fluent-bit/config# ls
containerd.lua  fluent-bit.conf  parsers.conf  parsers_multiline.conf  systemd.lua
root@fluent-bit-fnj2t:/fluent-bit/config# cat parsers.conf 
root@fluent-bit-fnj2t:/fluent-bit/config# cat parsers_multiline.conf
root@fluent-bit-fnj2t:/fluent-bit/config# 

It's due to https://github.com/fluent/fluent-operator/pull/1138, these two lines were added.