help request: [error] [multiline] parser 'java-multiline' not registered for 3.0.0 of operator

[meshpaul]

Describe the issue

We are seeing the following error in fluent-bit and then all fluent-bit daemons crash.

fluent operator version 3.0.0 fluent operator was installed via terraform pinned to chart version 3.0.0 and specifying provided values.yaml with fluent-bit enabled only

[2024/07/12 16:43:51] [ info] [input:emitter:emitter_for_multiline.4] storage_strategy='memory' (memory only)                                                                                                                                      [2024/07/12 16:43:51] [error] [multiline] parser 'java-multiline' not registered                                                                                                                                                                  
[2024/07/12 16:43:51] [error] Failed initialize filter multiline.4                                                                                                                                                                                 [2024/07/12 16:43:51] [error] [engine] filter initialization failed

the fluent values.yaml

fluentbit:

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*.log"
      skipLongLines: true
      readFromHead: false
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for tail input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
      #multiline: true
      #multiline.parser: "java-multiline"
      #multilineParser: "docker, cri, java"

  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline
    kubernetes:
      enable: true
      labels: true
      annotations: true
    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: false
## Omit other
  parsers:
    javaMultiline:
    # use in filter for parser generic springboot multiline log format
      enable: true

The custom chart related to Values.parsers.javaMultiline.enable or ClusterMultilineParser is created.

Does java-multiline parser requires additional configuration in 3.0.0 of fluent-operator in fluent-bit section of the config???

How did you install fluent operator?

terraform using helm resource pinning to chart version of 3.0.0 and specifying provided https://github.com/fluent/fluent-operator/blob/release-3.0/charts/fluent-operator/values.yaml

Additional context

No response

[cw-Guo]

can't reproduce using the lastest master.

helm upgrade -f charts/fluent-operator/values.yaml fluent-operator charts/fluent-operator

k get ClusterMultilineParser
NAME             AGE
java-multiline   3m32s

diff --git a/charts/fluent-operator/values.yaml b/charts/fluent-operator/values.yaml
index 1d1c17b..e9ea928 100644
--- a/charts/fluent-operator/values.yaml
+++ b/charts/fluent-operator/values.yaml
@@ -342,7 +342,7 @@ fluentbit:
   # The `filter` will filter and parse the collected log information and output the logs into a uniform format. You can choose whether to turn this on or not.
   filter:
     multiline:
-      enable: false
+      enable: true
       keyContent: log
       # emitterMemBufLimit 120 (MB)
       emitterMemBufLimit: 120
@@ -351,7 +351,7 @@ fluentbit:
         - python
         - java
         #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
-        # - java-multiline
+        - java-multiline
     kubernetes:
       enable: true
       labels: false
@@ -378,7 +378,7 @@ fluentbit:
   parsers:
     javaMultiline:
     # use in filter for parser generic springboot multiline log format
-      enable: false
+      enable: true

~/playground/fluent-operator master* ❯ kl -f fluent-bit-75jhg | grep filter                                                       21:45:08
[2024/07/13 04:43:32] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443
[2024/07/13 04:43:32] [ info] [filter:kubernetes:kubernetes.1]  token updated
[2024/07/13 04:43:32] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2024/07/13 04:43:32] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2024/07/13 04:43:32] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2024/07/13 04:43:32] [ info] [filter:multiline:multiline.5] created emitter: emitter_for_multiline.5
[2024/07/13 04:43:32] [ info] [filter:multiline:multiline.5] created new multiline stream for tail.1_kube.var.log.containers.fluent-bit-75jhg_fluent_fluent-bit-d1f9c97b2c80d953f506b94374b88d71bc1fc08ebd23058ec0aecd700610cc37.log
[2024/07/13 04:43:32] [ info] [filter:multiline:multiline.5] created new multiline stream for tail.1_kube.var.log.containers.kindnet-b7xv7_kube-system_kindnet-cni-188e0059e4fbbac149a995ce52e2dd463a626b5f2ea845d7d5d76f2dc620fdff.log
[2024/07/13 04:43:32] [ info] [filter:multiline:multiline.5] created new multiline stream for tail.1_kube.var.log.containers.kube-controller-manager-kind-control-plane_kube-system_kube-controller-manager-4c142639ac736cf7648bd6060943b37c5a1be2742b1c08558f4eaf38bb8fa496.log
[2024/07/13 04:44:11] [ info] [filter:multiline:multiline.5] created new multiline stream for tail.1_kube.var.log.containers.kube-apiserver-kind-control-plane_kube-system_kube-apiserver-31a4dd3e03f890b341d35ddc963658c3a253bbeb8824556223fa000041393595.log

[cw-Guo]

@meshpaul can you please share more details, like the values file you use?

[meshpaul]

@meshpaul can you please share more details, like the values file you use?

Yes will do thank you

[meshpaul]

@cw-Guo as requested. And thank you in advance. here is full values.yaml

# Default values for fluentbit-operator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Set this to containerd or crio if you want to collect CRI format logs
containerRuntime: containerd
#  If you want to deploy a default Fluent Bit pipeline (including Fluent Bit Input, Filter, and output) to collect Kubernetes logs, you'll need to set the Kubernetes parameter to true
# see https://github.com/fluent/fluent-operator/tree/master/manifests/logging-stack
Kubernetes: true

operator:
  # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
  # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
  initcontainer:
    repository: "docker"
    tag: "20.10"

    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  container:
    repository: "kubesphere/fluent-operator"
    tag: "v3.0.0"
  # nodeSelector configuration for Fluent Operator. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations: []
  # Priority class applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Pod security context for Fluent Operator. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Container security context for Fluent Operator container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # Fluent Operator resources. Usually user needn't to adjust these.
  resources:
    limits:
      cpu: 100m
      memory: 60Mi
    requests:
      cpu: 100m
      memory: 20Mi
  # Specify custom annotations to be added to each Fluent Operator pod.
  annotations: {}
  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  imagePullSecrets: []
  # - name: "image-pull-secret"
  # Reference one more key-value pairs of labels that should be attached to fluent-operator
  labels: {}
  #  myExampleLabel: someValue
  logPath:
    # The operator currently assumes a Docker container runtime path for the logs as the default, for other container runtimes you can set the location explicitly below.
    # crio: /var/log
    containerd: /var/log
  # By default, the operator provisions both Fluent Bit and FluentD controllers.
  # A specific controller can be disabled by setting the disableComponentControllers value.
  # The disableComponentControllers value can be either "fluent-bit" or "fluentd".
  # This helm chart renders the controllers CRDs in sub charts.
  # If needed a sub chart, hence corresponding set of CRDs can be disabled by
  # setting fluentbit.crdsEnable or fluentd.crdsEnable values to false.
  # By default all CRDs are deployed.
  disableComponentControllers: "fluentd"
  # Extra arguments given to the controller flags
  extraArgs: []
    # - --watch-namespaces=logging

fluentbit:
  # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
  crdsEnable: true
  enable: true
  serviceMonitor:
    enable: false
    interval: 30s
    path: /api/v2/metrics/prometheus
    scrapeTimeout: 10s
    secure: false
    tlsConfig: {}
    relabelings: []
    metricRelabelings: []
  image:
    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
    tag: "3.1.0"
  # fluentbit resources. If you do want to specify resources, adjust them as necessary
  # You can adjust it based on the log volume.
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 10m
      memory: 25Mi
  # Specify custom annotations to be added to each FluentBit pod.
  annotations:
    {}
    ## Request to Fluent Bit to exclude or not the logs generated by the Pod.
    # fluentbit.io/exclude: "true"
    ## Prometheus can use this tag to automatically discover the Pod and collect monitoring data
    # prometheus.io/scrape: "true"
  # Specify additional custom labels for fluentbit-pods
  labels: {}

  # Specify additional custom annotations for fluentbit-serviceaccount
  serviceAccountAnnotations: {}

  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  imagePullSecrets: []
  # - name: "image-pull-secret"
  logLevel: ""
  secrets: []
  # fluent-bit daemonset use host network
  hostNetwork: false
  # Pod security context for Fluent Bit pods. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Security context for Fluent Bit container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # List of volumes that can be mounted by containers belonging to the pod.
  additionalVolumes: []
  # Additional rbac rules which will be applied to the fluent-bit clusterrole. Ref: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding
  # NOTE: As fluent-bit is managed by the fluent-operator, fluent-bit can only be granted permissions the operator also has 
  rbacRules: {}
  # Pod volumes to mount into the container's filesystem.
  additionalVolumesMounts: []
  # affinity configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: DoesNotExist
  # nodeSelector configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations:
    - operator: Exists
  # Priority Class applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Environment variables that can be passed to fluentbit pods
  envVars: []
  #  - name: FOO
  #    value: "bar"
  schedulerName: ""

  namespaceFluentBitCfgSelector: {}

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log"
      skipLongLines: true
      readFromHead: false
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for tail input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
      #multiline: true
      #multiline.parser: "java-multiline"
      #multilineParser: "docker, cri, java"
    systemd:
      enable: false
      systemdFilter:
        enable: true
        filters: []
      path: "/var/log/journal"
      includeKubelet: true
      stripUnderscores: "off"
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for systemd input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
    nodeExporterMetrics: {}

  # Configure the output plugin parameter in FluentBit.
  # You can set enable to true to output logs to the specified location.
  output:
    #  You can find more supported output plugins here: https://github.com/fluent/fluent-operator/tree/master/docs/plugins/fluentbit/output
    stdout:
    # Use for debug to write collected logs to system out
      enable: true
      format: json
    # Uncomment the following section to enable Prometheus metrics exporter.
    prometheusMetricsExporter: {}

  # Configure the default filters in FluentBit.
  # The `filter` will filter and parse the collected log information and output the logs into a uniform format. You can choose whether to turn this on or not.
  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline
    kubernetes:
      enable: true
      labels: true
      annotations: true
    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: false
    systemd:
      enable: false

  kubeedge:
    enable: false

  # removes the hostPath mounts for varlibcontainers, varlogs and systemd.
  disableLogVolumes: false

  parsers:
    javaMultiline:
    # use in filter for parser generic springboot multiline log format
      enable: true

nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""

kubectl get ClusterMultilineParser
NAME             AGE
java-multiline   5d19h

[cw-Guo]

@cw-Guo as requested. And thank you in advance. here is full values.yaml

# Default values for fluentbit-operator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Set this to containerd or crio if you want to collect CRI format logs
containerRuntime: containerd
#  If you want to deploy a default Fluent Bit pipeline (including Fluent Bit Input, Filter, and output) to collect Kubernetes logs, you'll need to set the Kubernetes parameter to true
# see https://github.com/fluent/fluent-operator/tree/master/manifests/logging-stack
Kubernetes: true

operator:
  # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
  # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
  initcontainer:
    repository: "docker"
    tag: "20.10"

    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  container:
    repository: "kubesphere/fluent-operator"
    tag: "v3.0.0"
  # nodeSelector configuration for Fluent Operator. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations: []
  # Priority class applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Pod security context for Fluent Operator. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Container security context for Fluent Operator container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # Fluent Operator resources. Usually user needn't to adjust these.
  resources:
    limits:
      cpu: 100m
      memory: 60Mi
    requests:
      cpu: 100m
      memory: 20Mi
  # Specify custom annotations to be added to each Fluent Operator pod.
  annotations: {}
  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  imagePullSecrets: []
  # - name: "image-pull-secret"
  # Reference one more key-value pairs of labels that should be attached to fluent-operator
  labels: {}
  #  myExampleLabel: someValue
  logPath:
    # The operator currently assumes a Docker container runtime path for the logs as the default, for other container runtimes you can set the location explicitly below.
    # crio: /var/log
    containerd: /var/log
  # By default, the operator provisions both Fluent Bit and FluentD controllers.
  # A specific controller can be disabled by setting the disableComponentControllers value.
  # The disableComponentControllers value can be either "fluent-bit" or "fluentd".
  # This helm chart renders the controllers CRDs in sub charts.
  # If needed a sub chart, hence corresponding set of CRDs can be disabled by
  # setting fluentbit.crdsEnable or fluentd.crdsEnable values to false.
  # By default all CRDs are deployed.
  disableComponentControllers: "fluentd"
  # Extra arguments given to the controller flags
  extraArgs: []
    # - --watch-namespaces=logging

fluentbit:
  # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
  crdsEnable: true
  enable: true
  serviceMonitor:
    enable: false
    interval: 30s
    path: /api/v2/metrics/prometheus
    scrapeTimeout: 10s
    secure: false
    tlsConfig: {}
    relabelings: []
    metricRelabelings: []
  image:
    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
    tag: "3.1.0"
  # fluentbit resources. If you do want to specify resources, adjust them as necessary
  # You can adjust it based on the log volume.
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 10m
      memory: 25Mi
  # Specify custom annotations to be added to each FluentBit pod.
  annotations:
    {}
    ## Request to Fluent Bit to exclude or not the logs generated by the Pod.
    # fluentbit.io/exclude: "true"
    ## Prometheus can use this tag to automatically discover the Pod and collect monitoring data
    # prometheus.io/scrape: "true"
  # Specify additional custom labels for fluentbit-pods
  labels: {}

  # Specify additional custom annotations for fluentbit-serviceaccount
  serviceAccountAnnotations: {}

  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  imagePullSecrets: []
  # - name: "image-pull-secret"
  logLevel: ""
  secrets: []
  # fluent-bit daemonset use host network
  hostNetwork: false
  # Pod security context for Fluent Bit pods. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Security context for Fluent Bit container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # List of volumes that can be mounted by containers belonging to the pod.
  additionalVolumes: []
  # Additional rbac rules which will be applied to the fluent-bit clusterrole. Ref: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding
  # NOTE: As fluent-bit is managed by the fluent-operator, fluent-bit can only be granted permissions the operator also has 
  rbacRules: {}
  # Pod volumes to mount into the container's filesystem.
  additionalVolumesMounts: []
  # affinity configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: DoesNotExist
  # nodeSelector configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations:
    - operator: Exists
  # Priority Class applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Environment variables that can be passed to fluentbit pods
  envVars: []
  #  - name: FOO
  #    value: "bar"
  schedulerName: ""

  namespaceFluentBitCfgSelector: {}

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log"
      skipLongLines: true
      readFromHead: false
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for tail input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
      #multiline: true
      #multiline.parser: "java-multiline"
      #multilineParser: "docker, cri, java"
    systemd:
      enable: false
      systemdFilter:
        enable: true
        filters: []
      path: "/var/log/journal"
      includeKubelet: true
      stripUnderscores: "off"
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for systemd input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
    nodeExporterMetrics: {}

  # Configure the output plugin parameter in FluentBit.
  # You can set enable to true to output logs to the specified location.
  output:
    #  You can find more supported output plugins here: https://github.com/fluent/fluent-operator/tree/master/docs/plugins/fluentbit/output
    stdout:
    # Use for debug to write collected logs to system out
      enable: true
      format: json
    # Uncomment the following section to enable Prometheus metrics exporter.
    prometheusMetricsExporter: {}

  # Configure the default filters in FluentBit.
  # The `filter` will filter and parse the collected log information and output the logs into a uniform format. You can choose whether to turn this on or not.
  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline
    kubernetes:
      enable: true
      labels: true
      annotations: true
    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: false
    systemd:
      enable: false

  kubeedge:
    enable: false

  # removes the hostPath mounts for varlibcontainers, varlogs and systemd.
  disableLogVolumes: false

  parsers:
    javaMultiline:
    # use in filter for parser generic springboot multiline log format
      enable: true

nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""

kubectl get ClusterMultilineParser
NAME             AGE
java-multiline   5d19h

it still works

[2024/07/15 16:27:23] [ info] [fluent bit] version=3.1.0, commit=8ed2e80a9e, pid=17
[2024/07/15 16:27:23] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/07/15 16:27:23] [ info] [cmetrics] version=0.9.1
[2024/07/15 16:27:23] [ info] [ctraces ] version=0.5.1
[2024/07/15 16:27:23] [ info] [input:tail:tail.0] initializing
[2024/07/15 16:27:23] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/07/15 16:27:23] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/07/15 16:27:23] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2024/07/15 16:27:23] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2024/07/15 16:27:23] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/07/15 16:27:23] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/07/15 16:27:23] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/07/15 16:27:23] [ info] [filter:multiline:multiline.4] created emitter: emitter_for_multiline.4
[2024/07/15 16:27:23] [ info] [input:emitter:emitter_for_multiline.4] initializing
[2024/07/15 16:27:23] [ info] [input:emitter:emitter_for_multiline.4] storage_strategy='memory' (memory only)
[2024/07/15 16:27:23] [ info] [output:stdout:stdout.0] worker #0 started
[2024/07/15 16:27:23] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/07/15 16:27:23] [ info] [sp] stream processor started

[meshpaul]

Ok let me re-install using latest

[meshpaul]

Hi @cw-Guo So I was able to -clean up and re-run installation using Helm pointing to 3.0.0 helm chart release

 resource "helm_release" "fluent" {
  namespace = "fluent"
  name = "fluent"
  repository = "https://fluent.github.io/helm-charts"
  chart = "fluent-operator"
  values = [file("../helm-charts/fluent-operator/values.yaml")]
  version = "3.0.0"

  create_namespace = true
  set {
    name  = "fluentbit.nodeSelector.kubernetes\\.io/arch"
    value = "arm64"
  }
  set {
    name  = "operator.nodeSelector.kubernetes\\.io/arch"
    value = "arm64"
  }
}

The value chart is exactly the same as listed earlier... except I enabled containerd as follows:

    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: true

I didn't use latest as I pinned to the exact version and I have exactly the same result.

I suspect it may be due to my environment. EKS, K8s version v1.29.4-eks-036c24b, nodes are ARM based, and container runtime is containerd

But will try to run it once more using Latest... I am thinking to bump the version of fluent-bit to latest in values.yaml to match latest 3.1.2

Fluent-operator log

setenv + echo 'CONTAINER_ROOT_DIR=/var/log'                                                                                                                                                                                                                       │
│ fluent-operator 2024-07-16T20:43:01Z    INFO    controller-runtime.metrics    Metrics server is starting to listen    {"addr": ":8080"}                                                                                                                           │
│ fluent-operator 2024-07-16T20:43:01Z    INFO    setup    starting manager                                                                                                                                                                                         │
│ fluent-operator 2024-07-16T20:43:01Z    INFO    Starting server    {"path": "/metrics", "kind": "metrics", "addr": ":8080"}                                                                                                                                       │
│ fluent-operator 2024-07-16T20:43:01Z    INFO    Starting server    {"kind": "health probe", "addr": ":8081"}                                                                                                                                                      │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBit"}                                        │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.ServiceAccount"}                                         │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.DaemonSet"}                                              │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting Controller    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit"}                                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1alpha2.Collector"}                                        │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.ServiceAccount"}                                         │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.StatefulSet"}                                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "source": "kind source: *v1.Service"}                                                │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting Controller    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector"}                                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBit"}                                        │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1.Secret"}                                                 │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterFluentBitConfig"}                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.FluentBitConfig"}                                  │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterInput"}                                     │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterFilter"}                                    │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterOutput"}                                    │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterParser"}                                    │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.ClusterMultilineParser"}                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Filter"}                                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Output"}                                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.Parser"}                                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting EventSource    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "source": "kind source: *v1alpha2.MultilineParser"}                                  │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting Controller    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit"}                                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting workers    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "worker count": 1}                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting workers    {"controller": "collector", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "Collector", "worker count": 1}                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    Starting workers    {"controller": "fluentbit", "controllerGroup": "fluentbit.fluent.io", "controllerKind": "FluentBit", "worker count": 1}                                                                       │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    KubeAPIWarningLogger    metadata.finalizers: "fluentbit.fluent.io": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers                                           │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ fluent-operator 2024-07-16T20:43:02Z    INFO    controllers.FluentBitConfig    Fluent Bit main configuration has updated    {"logging-control-plane": "fluent", "fluentbitconfig": "fluent-bit-config", "secret": "fluent-bit-config"}                            │
│ Stream closed EOF for fluent/fluent-operator-5fff756565-bcb26 (setenv)

The fluent-bit pods are all down with same errors

│ level=info time=2024-07-16T20:54:07Z msg="No fluent-bit secret yaml found, using classic one."                                                                                                                                                                    │
│ level=info time=2024-07-16T20:54:07Z msg="fluent-bit started"                                                                                                                                                                                                     │
│ Fluent Bit v3.1.0                                                                                                                                                                                                                                                 │
│ * Copyright (C) 2015-2024 The Fluent Bit Authors                                                                                                                                                                                                                  │
│ * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd                                                                                                                                                                                                  │
│ * https://fluentbit.io                                                                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                   │
│ ______ _                  _    ______ _ _           _____  __                                                                                                                                                                                                     │
│ |  ___| |                | |   | ___ (_) |         |____ |/  |                                                                                                                                                                                                    │
│ | |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |                                                                                                                                                                                                    │
│ |  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |                                                                                                                                                                                                    │
│ | |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_                                                                                                                                                                                                   │
│ \_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/                                                                                                                                                                                                   │
│                                                                                                                                                                                                                                                                   │
│ [2024/07/16 20:54:07] [ info] [fluent bit] version=3.1.0, commit=8ed2e80a9e, pid=12                                                                                                                                                                               │
│ [2024/07/16 20:54:07] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128                                                                                                                                                      │
│ [2024/07/16 20:54:07] [ info] [cmetrics] version=0.9.1                                                                                                                                                                                                            │
│ [2024/07/16 20:54:07] [ info] [ctraces ] version=0.5.1                                                                                                                                                                                                            │
│ [2024/07/16 20:54:07] [ info] [input:tail:tail.0] initializing                                                                                                                                                                                                    │
│ [2024/07/16 20:54:07] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)                                                                                                                                                                         │
│ [2024/07/16 20:54:07] [error] [input:tail:tail.0] parser 'cri' is not registered                                                                                                                                                                                  │
│ [2024/07/16 20:54:07] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0                                                                                                                                                  │
│ [2024/07/16 20:54:07] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443                                                                                                                                                       │
│ [2024/07/16 20:54:07] [ info] [filter:kubernetes:kubernetes.1]  token updated                                                                                                                                                                                     │
│ [2024/07/16 20:54:07] [ info] [filter:kubernetes:kubernetes.1] local POD info OK                                                                                                                                                                                  │
│ [2024/07/16 20:54:07] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...                                                                                                                                                            │
│ [2024/07/16 20:54:07] [ info] [filter:kubernetes:kubernetes.1] connectivity OK                                                                                                                                                                                    │
│ [2024/07/16 20:54:07] [ info] [filter:multiline:multiline.5] created emitter: emitter_for_multiline.5                                                                                                                                                             │
│ [2024/07/16 20:54:07] [ info] [input:emitter:emitter_for_multiline.5] initializing                                                                                                                                                                                │
│ [2024/07/16 20:54:07] [ info] [input:emitter:emitter_for_multiline.5] storage_strategy='memory' (memory only)                                                                                                                                                     │
│ [2024/07/16 20:54:07] [error] [multiline] parser 'java-multiline' not registered                                                                                                                                                                                  │
│ [2024/07/16 20:54:07] [error] Failed initialize filter multiline.5                                                                                                                                                                                                │
│ [2024/07/16 20:54:07] [error] [engine] filter initialization failed                                                                                                                                                                                               │
│ [2024/07/16 20:54:08] [ info] [input] pausing tail.0                                                                                                                                                                                                              │
│ [2024/07/16 20:54:08] [ info] [input] pausing emitter_for_multiline.5                                                                                                                                                                                             │
│ level=error time=2024-07-16T20:54:08Z msg="Failure during the run time of fluent-bit" error="failed to run fluent-bit: exit status 255"                                                                                                                           │
│ Stream closed EOF for fluent/fluent-bit-qvjzn (fluent-bit)

kubectl get ClusterFilter NAME AGE containerd 30m kubernetes 30m multiline 30m

[benjaminhuo]

I suspect it may be due to my environment. EKS, K8s version v1.29.4-eks-036c24b, nodes are ARM based, and container runtime is containerd

what else should we do to run fluentbit on arm? @wanjunlei

[wanjunlei]

parser 'cri' is not registered

The error parser 'cri' not registered may be caused by the parser file not being found, so change the fluentbit image to a debug image, then exec into the fluenbit pod and make sure /fluent-bit/etc/parser.conf exists and contains the cri parser.

[meshpaul]

parser 'cri' is not registered

The error parser 'cri' not registered may be caused by the parser file not being found, so change the fluentbit image to a debug image, then exec into the fluenbit pod and make sure /fluent-bit/etc/parser.conf exists and contains the cri parser.

I'll be happy to do debugging. But I am new to this... any refs will be appreciated.

[meshpaul]

FYI, the latest run I enabled containerd in case it wasn't clear in values yaml

Changed settings

    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: true

Latest values yaml just in case

# Default values for fluentbit-operator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Set this to containerd or crio if you want to collect CRI format logs
containerRuntime: containerd
#  If you want to deploy a default Fluent Bit pipeline (including Fluent Bit Input, Filter, and output) to collect Kubernetes logs, you'll need to set the Kubernetes parameter to true
# see https://github.com/fluent/fluent-operator/tree/master/manifests/logging-stack
Kubernetes: true

operator:
  # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
  # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
  initcontainer:
    repository: "docker"
    tag: "20.10"

    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
  container:
    repository: "kubesphere/fluent-operator"
    tag: "v3.0.0"
  # nodeSelector configuration for Fluent Operator. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations: []
  # Priority class applied to Fluent Operator. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Pod security context for Fluent Operator. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Container security context for Fluent Operator container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # Fluent Operator resources. Usually user needn't to adjust these.
  resources:
    limits:
      cpu: 100m
      memory: 60Mi
    requests:
      cpu: 100m
      memory: 20Mi
  # Specify custom annotations to be added to each Fluent Operator pod.
  annotations: {}
  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  imagePullSecrets: []
  # - name: "image-pull-secret"
  # Reference one more key-value pairs of labels that should be attached to fluent-operator
  labels: {}
  #  myExampleLabel: someValue
  logPath:
    # The operator currently assumes a Docker container runtime path for the logs as the default, for other container runtimes you can set the location explicitly below.
    # crio: /var/log
    containerd: /var/log
  # By default, the operator provisions both Fluent Bit and FluentD controllers.
  # A specific controller can be disabled by setting the disableComponentControllers value.
  # The disableComponentControllers value can be either "fluent-bit" or "fluentd".
  # This helm chart renders the controllers CRDs in sub charts.
  # If needed a sub chart, hence corresponding set of CRDs can be disabled by
  # setting fluentbit.crdsEnable or fluentd.crdsEnable values to false.
  # By default all CRDs are deployed.
  disableComponentControllers: "fluentd"
  # Extra arguments given to the controller flags
  extraArgs: []
    # - --watch-namespaces=logging

fluentbit:
  # Installs a sub chart carrying the CRDs for the fluent-bit controller. The sub chart is enabled by default.
  crdsEnable: true
  enable: true
  serviceMonitor:
    enable: false
    interval: 30s
    path: /api/v2/metrics/prometheus
    scrapeTimeout: 10s
    secure: false
    tlsConfig: {}
    relabelings: []
    metricRelabelings: []
  image:
    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
    tag: "3.1.0"
  # fluentbit resources. If you do want to specify resources, adjust them as necessary
  # You can adjust it based on the log volume.
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 10m
      memory: 25Mi
  # Specify custom annotations to be added to each FluentBit pod.
  annotations:
    {}
    ## Request to Fluent Bit to exclude or not the logs generated by the Pod.
    # fluentbit.io/exclude: "true"
    ## Prometheus can use this tag to automatically discover the Pod and collect monitoring data
    # prometheus.io/scrape: "true"
  # Specify additional custom labels for fluentbit-pods
  labels: {}

  # Specify additional custom annotations for fluentbit-serviceaccount
  serviceAccountAnnotations: {}

  ## Reference to one or more secrets to be used when pulling images
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  imagePullSecrets: []
  # - name: "image-pull-secret"
  logLevel: ""
  secrets: []
  # fluent-bit daemonset use host network
  hostNetwork: false
  # Pod security context for Fluent Bit pods. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  podSecurityContext: {}
  # Security context for Fluent Bit container. Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext: {}
  # List of volumes that can be mounted by containers belonging to the pod.
  additionalVolumes: []
  # Additional rbac rules which will be applied to the fluent-bit clusterrole. Ref: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding
  # NOTE: As fluent-bit is managed by the fluent-operator, fluent-bit can only be granted permissions the operator also has 
  rbacRules: {}
  # Pod volumes to mount into the container's filesystem.
  additionalVolumesMounts: []
  # affinity configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: DoesNotExist
  # nodeSelector configuration for Fluent Bit pods. Ref: https://kubernetes.io/docs/user-guide/node-selection/
  nodeSelector: {}
  # Node tolerations applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  tolerations:
    - operator: Exists
  # Priority Class applied to Fluent Bit pods. Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""
  # Environment variables that can be passed to fluentbit pods
  envVars: []
  #  - name: FOO
  #    value: "bar"
  schedulerName: ""

  # Remove the above empty volumes and volumesMounts, and then set additionalVolumes and additionalVolumesMounts as below if you want to collect node exporter metrics
  # additionalVolumes:
  #   - name: hostProc
  #     hostPath:
  #       path: /proc/
  #   - name: hostSys
  #     hostPath:
  #       path: /sys/
  # Uncomment the code if you intend to create the volume for buffer storage in case the storage type "filesystem" is being used in the configuration of the fluentbit service.
  #   - name: hostBuffer
  #     hostPath:
  #       path: /tmp/fluent-bit-buffer
  # additionalVolumesMounts:
  #   - mountPath: /host/sys
  #     mountPropagation: HostToContainer
  #     name: hostSys
  #     readOnly: true
  #   - mountPath: /host/proc
  #     mountPropagation: HostToContainer
  #     name: hostProc
  #     readOnly: true
  # Uncomment the code if you intend to mount the volume for buffer storage in case the storage type "filesystem" is being used in the configuration of the fluentbit service.
  #   - mountPath: /host/fluent-bit-buffer
  #     mountPropagation: HostToContainer
  #     name: hostBuffer

  namespaceFluentBitCfgSelector: {}

  # Set a limit of memory that Tail plugin can use when appending data to the Engine.
  # You can find more details here: https://docs.fluentbit.io/manual/pipeline/inputs/tail#config
  # If the limit is reach, it will be paused; when the data is flushed it resumes.
  # if the inbound traffic is less than 2.4Mbps, setting memBufLimit to 5MB is enough
  # if the inbound traffic is less than 4.0Mbps, setting memBufLimit to 10MB is enough
  # if the inbound traffic is less than 13.64Mbps, setting memBufLimit to 50MB is enough
  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      #path: "/var/log/containers/*.log"
      path: "/var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log, /var/log/containers/*_keycloak_*.log, /var/log/containers/*_database_*.log "
      skipLongLines: true
      readFromHead: false
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for tail input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
      #multiline: true
      #multiline.parser: "java-multiline"
      #multilineParser: "docker, cri, java"
    systemd:
      enable: false
      systemdFilter:
        enable: true
        filters: []
      path: "/var/log/journal"
      includeKubelet: true
      stripUnderscores: "off"
      # Use storageType as "filesystem" if you want to use filesystem as the buffering mechanism for systemd input.
      storageType: memory
      pauseOnChunksOverlimit: "off"
    nodeExporterMetrics: {}
    # uncomment below nodeExporterMetrics section if you want to collect node exporter metrics
  #   nodeExporterMetrics:
  #     tag: node_metrics
  #     scrapeInterval: 15s
  #     path:
  #       procfs: /host/proc
  #       sysfs: /host/sys
    fluentBitMetrics: {}
    # uncomment below fluentBitMetrics section if you want to collect fluentBit metrics
#    fluentBitMetrics:
#      scrapeInterval: "2"
#      scrapeOnStart: true
#      tag: "fb.metrics"

  # Configure the output plugin parameter in FluentBit.
  # You can set enable to true to output logs to the specified location.
  output:
    #  You can find more supported output plugins here: https://github.com/fluent/fluent-operator/tree/master/docs/plugins/fluentbit/output
    stdout:
    # Use for debug to write collected logs to system out
      enable: true
      format: json
    # Uncomment the following section to enable Prometheus metrics exporter.
    prometheusMetricsExporter: {}
#    prometheusMetricsExporter:
#      match: "fb.metrics"
#      metricsExporter:
#        host: "0.0.0.0"
#        port: 2020
#        addLabels:
#          app: "fluentbit"

  # Configure the default filters in FluentBit.
  # The `filter` will filter and parse the collected log information and output the logs into a uniform format. You can choose whether to turn this on or not.
  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline
    kubernetes:
      enable: true
      labels: true
      annotations: true
    containerd:
      # This is customized lua containerd log format converter, you can refer here:
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterfilter-containerd.yaml
      # https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-containerd-config.yaml
      enable: true
    systemd:
      enable: false

  kubeedge:
    enable: false
    prometheusRemoteWrite:
      # Change the host to the address of a cloud-side Prometheus-compatible server that can receive Prometheus remote write data
      host: "<cloud-prometheus-service-host>"
      # Change the port to the port of a cloud-side Prometheus-compatible server that can receive Prometheus remote write data
      port: "<cloud-prometheus-service-port>"

  # removes the hostPath mounts for varlibcontainers, varlogs and systemd.
  disableLogVolumes: false

  parsers:
    javaMultiline:
    # use in filter for parser generic springboot multiline log format
      enable: true

nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""

[meshpaul]

parser 'cri' is not registered

The error parser 'cri' not registered may be caused by the parser file not being found, so change the fluentbit image to a debug image, then exec into the fluenbit pod and make sure /fluent-bit/etc/parser.conf exists and contains the cri parser.

Hi I have updated yaml to include debug image

    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
    tag: "3.1.0-debug"

I do see /fluent-bit/etc/parser.conf exists and it contains the cri parser. and I still get the same error

[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

Here is log output without java-multiline enabled

 Fluent Bit v3.1.0                                                                                                                                     │
│ * Copyright (C) 2015-2024 The Fluent Bit Authors                                                                                                      │
│ * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd                                                                                      │
│ * https://fluentbit.io                                                                                                                                │
│                                                                                                                                                       │
│ ______ _                  _    ______ _ _           _____  __                                                                                         │
│ |  ___| |                | |   | ___ (_) |         |____ |/  |                                                                                        │
│ | |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |                                                                                        │
│ |  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |                                                                                        │
│ | |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_                                                                                       │
│ \_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/                                                                                       │
│                                                                                                                                                       │
│ level=info time=2024-07-17T18:13:59Z msg="No fluent-bit secret yaml found, using classic one."                                                        │
│ level=info time=2024-07-17T18:13:59Z msg="fluent-bit started"                                                                                         │
│ [2024/07/17 18:13:59] [ info] [fluent bit] version=3.1.0, commit=8ed2e80a9e, pid=12                                                                   │
│ [2024/07/17 18:13:59] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128                                          │
│ [2024/07/17 18:13:59] [ info] [cmetrics] version=0.9.1                                                                                                │
│ [2024/07/17 18:13:59] [ info] [ctraces ] version=0.5.1                                                                                                │
│ [2024/07/17 18:13:59] [ info] [input:tail:tail.0] initializing                                                                                        │
│ [2024/07/17 18:13:59] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)                                                             │
│ [2024/07/17 18:13:59] [error] [input:tail:tail.0] parser 'cri' is not registered                                                                      │
│ [2024/07/17 18:13:59] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0                                      │
│ [2024/07/17 18:13:59] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443                                           │
│ [2024/07/17 18:13:59] [ info] [filter:kubernetes:kubernetes.1]  token updated                                                                         │
│ [2024/07/17 18:13:59] [ info] [filter:kubernetes:kubernetes.1] local POD info OK                                                                      │
│ [2024/07/17 18:13:59] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...                                                │
│ [2024/07/17 18:14:00] [ info] [filter:kubernetes:kubernetes.1] connectivity OK                                                                        │
│ [2024/07/17 18:14:00] [ info] [filter:multiline:multiline.5] created emitter: emitter_for_multiline.5                                                 │
│ [2024/07/17 18:14:00] [ info] [input:emitter:emitter_for_multiline.5] initializing                                                                    │
│ [2024/07/17 18:14:00] [ info] [input:emitter:emitter_for_multiline.5] storage_strategy='memory' (memory only)                                         │
│ [2024/07/17 18:14:00] [ info] [output:null:null.0] worker #0 started                                                                                  │
│ [2024/07/17 18:14:00] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020                                                                        │
│ [2024/07/17 18:14:00] [ info] [sp] stream processor started                                                                                           │
│ [2024/07/17 18:14:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=20101933 watch_fd=1 name=/var/log/containers/portal-service-76b8d77c64-rr5d │
│ [2024/07/17 18:14:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=20101930 watch_fd=2 name=/var/log/containers/portal-service-76b8d77c64-rr5d │
│ [2024/07/17 18:14:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=41112997 watch_fd=3 name=/var/log/containers/utility-rules-28685520-lv229_d │
│ [2024/07/17 18:14:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=14057149 watch_fd=4 name=/var/log/containers/utility-rules-28685520-rcdbw_s │
│ [2024/07/17 18:14:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=40943536 watch_fd=5 name=/var/log/containers/utility-rules-28686960-8p78m_s │
│

[meshpaul]

Another shot in the dark changed fluent-bit to 3.1.2 same result...

│ ______ _                  _    ______ _ _           _____  __                                                                                         │
│ |  ___| |                | |   | ___ (_) |         |____ |/  |                                                                                        │
│ | |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |                                                                                        │
│ |  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |                                                                                        │
│ | |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_                                                                                       │
│ \_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/                                                                                       │
│                                                                                                                                                       │
│ [2024/07/17 19:45:47] [ info] [fluent bit] version=3.1.2, commit=a6feacd6e9, pid=12                                                                   │
│ [2024/07/17 19:45:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128                                          │
│ [2024/07/17 19:45:47] [ info] [cmetrics] version=0.9.1                                                                                                │
│ [2024/07/17 19:45:47] [ info] [ctraces ] version=0.5.1                                                                                                │
│ [2024/07/17 19:45:47] [ info] [input:tail:tail.0] initializing                                                                                        │
│ [2024/07/17 19:45:47] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)                                                             │
│ [2024/07/17 19:45:47] [error] [input:tail:tail.0] parser 'cri' is not registered

[wanjunlei]

Please provide the yaml file of secret fluent-bit-config.

[meshpaul]

@wanjunlei as requested below secrets in YAML and other formats kubectl -n fluent get secret fluent-bit-config -o yaml. Fluent bit image is 3.1.0-debug

parsers set to following in values.yaml

  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        #- java-multiline

I have several outputs , encoded via yaml and decoded below kubectl -n fluent get secret fluent-bit-config -o yaml

apiVersion: v1
data:
  containerd.lua: ZnVuY3Rpb24gY29udGFpbmVyZCggdGFnLCB0aW1lc3RhbXAsIHJlY29yZCkKICAgICAgIGlmKHJlY29yZFsibG9ndGFnIl1+PW5pbCkKICAgICAgIHRoZW4KICAgICAgIHRpbWVTdHIgPSBvcy5kYXRlKCIhKnQiLCAgdGltZXN0YW1wWyJzZWMiXSkKICAgICAgICB0ID0gc3RyaW5nLmZvcm1hdCgiJTRkLSUwMmQtJTAyZFQlMDJkOiUwMmQ6JTAyZC4lc1oiLAogICAgICAgIHRpbWVTdHJbInllYXIiXSwgdGltZVN0clsibW9udGgiXSwgdGltZVN0clsiZGF5Il0sCiAgICAgICAgdGltZVN0clsiaG91ciJdLCB0aW1lU3RyWyJtaW4iXSwgdGltZVN0clsic2VjIl0sCiAgICAgICAgdGltZXN0YW1wWyJuc2VjIl0pOwogICAgICAgIHJlY29yZFsidGltZSJdID0gdDsKICAgICAgICByZWNvcmRbImxvZyJdID0gcmVjb3JkWyJtZXNzYWdlIl07CiAgICAgICAgcmVjb3JkWyJtZXNzYWdlIl0gPSAgbmlsOwogICAgICAgIHJldHVybiAxLCB0aW1lc3RhbXAsIHJlY29yZAogICAgICAgIGVsc2UKICAgICAgICByZXR1cm4gMCx0aW1lc3RhbXAscmVjb3JkCiAgICAgICBlbmQKZW5k
  fluent-bit.conf: W1NlcnZpY2VdCiAgICBIdHRwX1NlcnZlciAgICB0cnVlCltJbnB1dF0KICAgIE5hbWUgICAgdGFpbAogICAgUGF0aCAgICAvdmFyL2xvZy9jb250YWluZXJzLypfZGV2XyoubG9nLCAvdmFyL2xvZy9jb250YWluZXJzLypfc3RnXyoubG9nLCAvdmFyL2xvZy9jb250YWluZXJzLypfa2V5Y2xvYWtfKi5sb2csIC92YXIvbG9nL2NvbnRhaW5lcnMvKl9kYXRhYmFzZV8qLmxvZwogICAgUmVhZF9mcm9tX0hlYWQgICAgZmFsc2UKICAgIFJlZnJlc2hfSW50ZXJ2YWwgICAgMTAKICAgIFNraXBfTG9uZ19MaW5lcyAgICB0cnVlCiAgICBEQiAgICAvZmx1ZW50LWJpdC90YWlsL3Bvcy5kYgogICAgREIuU3luYyAgICBOb3JtYWwKICAgIE1lbV9CdWZfTGltaXQgICAgMTAwTUIKICAgIFBhcnNlciAgICBjcmkKICAgIFRhZyAgICBrdWJlLioKICAgIHN0b3JhZ2UudHlwZSAgICBtZW1vcnkKW0ZpbHRlcl0KICAgIE5hbWUgICAgbHVhCiAgICBNYXRjaCAgICBrdWJlLioKICAgIHNjcmlwdCAgICAvZmx1ZW50LWJpdC9jb25maWcvY29udGFpbmVyZC5sdWEKICAgIGNhbGwgICAgY29udGFpbmVyZAogICAgdGltZV9hc190YWJsZSAgICB0cnVlCltGaWx0ZXJdCiAgICBOYW1lICAgIGt1YmVybmV0ZXMKICAgIE1hdGNoICAgIGt1YmUuKgogICAgS3ViZV9VUkwgICAgaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjOjQ0MwogICAgS3ViZV9DQV9GaWxlICAgIC92YXIvcnVuL3NlY3JldHMva3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9jYS5jcnQKICAgIEt1YmVfVG9rZW5fRmlsZSAgICAvdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvdG9rZW4KICAgIEs4Uy1Mb2dnaW5nLkV4Y2x1ZGUgICAgdHJ1ZQogICAgTGFiZWxzICAgIHRydWUKICAgIEFubm90YXRpb25zICAgIHRydWUKW0ZpbHRlcl0KICAgIE5hbWUgICAgbmVzdAogICAgTWF0Y2ggICAga3ViZS4qCiAgICBPcGVyYXRpb24gICAgbGlmdAogICAgTmVzdGVkX3VuZGVyICAgIGt1YmVybmV0ZXMKICAgIEFkZF9wcmVmaXggICAga3ViZXJuZXRlc18KW0ZpbHRlcl0KICAgIE5hbWUgICAgbW9kaWZ5CiAgICBNYXRjaCAgICBrdWJlLioKICAgIFJlbW92ZSAgICBzdHJlYW0KICAgIFJlbW92ZSAgICBrdWJlcm5ldGVzX3BvZF9pZAogICAgUmVtb3ZlICAgIGt1YmVybmV0ZXNfaG9zdAogICAgUmVtb3ZlICAgIGt1YmVybmV0ZXNfY29udGFpbmVyX2hhc2gKW0ZpbHRlcl0KICAgIE5hbWUgICAgbmVzdAogICAgTWF0Y2ggICAga3ViZS4qCiAgICBPcGVyYXRpb24gICAgbmVzdAogICAgV2lsZGNhcmQgICAga3ViZXJuZXRlc18qCiAgICBOZXN0X3VuZGVyICAgIGt1YmVybmV0ZXMKICAgIFJlbW92ZV9wcmVmaXggICAga3ViZXJuZXRlc18KW0ZpbHRlcl0KICAgIE5hbWUgICAgbXVsdGlsaW5lCiAgICBNYXRjaCAgICBrdWJlLioKICAgIG11bHRpbGluZS5wYXJzZXIgICAgZ28scHl0aG9uLGphdmEsY3JpCiAgICBtdWx0aWxpbmUua2V5X2NvbnRlbnQgICAgbG9nCltPdXRwdXRdCiAgICBOYW1lICAgIHN0ZG91dAogICAgTWF0Y2ggICAgKgo=
  parsers.conf: ""
  parsers_multiline.conf: W01VTFRJTElORV9QQVJTRVJdCiAgICBOYW1lICAgIGphdmEtbXVsdGlsaW5lCiAgICBUeXBlICAgIHJlZ2V4CiAgICBLZXlfQ29udGVudCAgICBsb2cKICAgIEZsdXNoX1RpbWVvdXQgICAgMTAwMAogICAgUnVsZSAgICAic3RhcnRfc3RhdGUiICIvXlxbPyhcZCtcLVxkK1wtXGQrIFxkK1w6XGQrXDpcZCsoXC5cZCspPylcXT8gLyIgImNvbnQiCiAgICBSdWxlICAgICJjb250IiAiL15jb21cLi4qLyIgImNvbnQiCiAgICBSdWxlICAgICJjb250IiAiL15ccysuKi8iICJjb250IgogICAgUnVsZSAgICAiY29udCIgIi9eQ2F1c2VkLiokLyIgImNvbnQiCg==
kind: Secret
metadata:
  creationTimestamp: "2024-07-16T20:43:02Z"
  name: fluent-bit-config
  namespace: fluent
  ownerReferences:
  - apiVersion: fluentbit.fluent.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: ClusterFluentBitConfig
    name: fluent-bit-config
    uid: c84eb69d-78c3-4add-891b-18fd9b6c5261
  resourceVersion: "97546341"
  uid: ef432051-3e50-4ccd-85e1-06833439bcfc
type: Opaque

Output as raw text from secrets found in fluent-bit-config

containerd.lua [function containerd( tag, timestamp, record)
       if(record["logtag"]~=nil)
       then
       timeStr = os.date("!*t",  timestamp["sec"])
        t = string.format("%4d-%02d-%02dT%02d:%02d:%02d.%sZ",
        timeStr["year"], timeStr["month"], timeStr["day"],
        timeStr["hour"], timeStr["min"], timeStr["sec"],
        timestamp["nsec"]);
        record["time"] = t;
        record["log"] = record["message"];
        record["message"] =  nil;
        return 1, timestamp, record
        else
        return 0,timestamp,record
       end
end]
fluent-bit.conf [[Service]
    Http_Server    true
[Input]
    Name    tail
    Path    /var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log, /var/log/containers/*_keycloak_*.log, /var/log/containers/*_database_*.log
    Read_from_Head    false
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    100MB
    Parser    cri
    Tag    kube.*
    storage.type    memory
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    K8S-Logging.Exclude    true
    Labels    true
    Annotations    true
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    multiline
    Match    kube.*
    multiline.parser    go,python,java,cri
    multiline.key_content    log
[Output]
    Name    stdout
    Match    *
]
parsers.conf []
parsers_multiline.conf [[MULTILINE_PARSER]
    Name    java-multiline
    Type    regex
    Key_Content    log
    Flush_Timeout    1000
    Rule    "start_state" "/^\[?(\d+\-\d+\-\d+ \d+\:\d+\:\d+(\.\d+)?)\]? /" "cont"
    Rule    "cont" "/^com\..*/" "cont"
    Rule    "cont" "/^\s+.*/" "cont"
    Rule    "cont" "/^Caused.*$/" "cont"
]

output as JSON kubectl -n fluent get secret fluent-bit-config -o json | jq '.data' | jq 'map_values(@base64d)'

{
  "containerd.lua": "function containerd( tag, timestamp, record)\n       if(record[\"logtag\"]~=nil)\n       then\n       timeStr = os.date(\"!*t\",  timestamp[\"sec\"])\n        t = string.format(\"%4d-%02d-%02dT%02d:%02d:%02d.%sZ\",\n        timeStr[\"year\"], timeStr[\"month\"], timeStr[\"day\"],\n        timeStr[\"hour\"], timeStr[\"min\"], timeStr[\"sec\"],\n        timestamp[\"nsec\"]);\n        record[\"time\"] = t;\n        record[\"log\"] = record[\"message\"];\n        record[\"message\"] =  nil;\n        return 1, timestamp, record\n        else\n        return 0,timestamp,record\n       end\nend",
  "fluent-bit.conf": "[Service]\n    Http_Server    true\n[Input]\n    Name    tail\n    Path    /var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log, /var/log/containers/*_keycloak_*.log, /var/log/containers/*_database_*.log\n    Read_from_Head    false\n    Refresh_Interval    10\n    Skip_Long_Lines    true\n    DB    /fluent-bit/tail/pos.db\n    DB.Sync    Normal\n    Mem_Buf_Limit    100MB\n    Parser    cri\n    Tag    kube.*\n    storage.type    memory\n[Filter]\n    Name    lua\n    Match    kube.*\n    script    /fluent-bit/config/containerd.lua\n    call    containerd\n    time_as_table    true\n[Filter]\n    Name    kubernetes\n    Match    kube.*\n    Kube_URL    https://kubernetes.default.svc:443\n    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token\n    K8S-Logging.Exclude    true\n    Labels    true\n    Annotations    true\n[Filter]\n    Name    nest\n    Match    kube.*\n    Operation    lift\n    Nested_under    kubernetes\n    Add_prefix    kubernetes_\n[Filter]\n    Name    modify\n    Match    kube.*\n    Remove    stream\n    Remove    kubernetes_pod_id\n    Remove    kubernetes_host\n    Remove    kubernetes_container_hash\n[Filter]\n    Name    nest\n    Match    kube.*\n    Operation    nest\n    Wildcard    kubernetes_*\n    Nest_under    kubernetes\n    Remove_prefix    kubernetes_\n[Filter]\n    Name    multiline\n    Match    kube.*\n    multiline.parser    go,python,java,cri\n    multiline.key_content    log\n[Output]\n    Name    stdout\n    Match    *\n",
  "parsers.conf": "",
  "parsers_multiline.conf": "[MULTILINE_PARSER]\n    Name    java-multiline\n    Type    regex\n    Key_Content    log\n    Flush_Timeout    1000\n    Rule    \"start_state\" \"/^\\[?(\\d+\\-\\d+\\-\\d+ \\d+\\:\\d+\\:\\d+(\\.\\d+)?)\\]? /\" \"cont\"\n    Rule    \"cont\" \"/^com\\..*/\" \"cont\"\n    Rule    \"cont\" \"/^\\s+.*/\" \"cont\"\n    Rule    \"cont\" \"/^Caused.*$/\" \"cont\"\n"
}

Let me know if anything else is needed.

[wanjunlei]

The correct config file is this. Your config file misses the parser_file field.

[Service]
    Http_Server    true
    Parsers_File    /fluent-bit/etc/parsers.conf
    Parsers_File    /fluent-bit/config/parsers_multiline.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/systemd.db
    DB.Sync    Normal
    Tag    service.*
    Systemd_Filter    _SYSTEMD_UNIT=docker.service
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
    Strip_Underscores    off
    storage.type    memory
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Read_from_Head    false
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    100MB
    Parser    docker
    Tag    kube.*
    storage.type    memory
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    multiline
    Match    kube.*
    multiline.parser    go,python,java,java-multiline
    multiline.key_content    log
    flush_ms    2000
    emitter_storage.type    memory
    emitter_mem_buf_limit    120MB
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    null
    Match   *

This field is set by the clusterfluentbitconfigs.fluentbit.fluent.io.

kubectl get clusterfluentbitconfigs.fluentbit.fluent.io fluent-bit-config -oyaml
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-operator
    meta.helm.sh/release-namespace: fluent
  creationTimestamp: "2024-07-17T02:34:51Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit-config
  resourceVersion: "10179094"
  uid: bf63780e-f12f-411b-9ec0-e445f0cd6cad
spec:
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  multilineParserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  parserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    httpServer: true
    parsersFiles:
    - /fluent-bit/etc/parsers.conf
    - /fluent-bit/config/parsers_multiline.conf

[meshpaul]

The correct config file is this. Your config file misses the parser_file field.

[Service]
    Http_Server    true
    Parsers_File    /fluent-bit/etc/parsers.conf
    Parsers_File    /fluent-bit/config/parsers_multiline.conf
[Input]
    Name    systemd
    Path    /var/log/journal
    DB    /fluent-bit/tail/systemd.db
    DB.Sync    Normal
    Tag    service.*
    Systemd_Filter    _SYSTEMD_UNIT=docker.service
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
    Strip_Underscores    off
    storage.type    memory
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Read_from_Head    false
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    100MB
    Parser    docker
    Tag    kube.*
    storage.type    memory
[Filter]
    Name    lua
    Match    kube.*
    script    /fluent-bit/config/containerd.lua
    call    containerd
    time_as_table    true
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    multiline
    Match    kube.*
    multiline.parser    go,python,java,java-multiline
    multiline.key_content    log
    flush_ms    2000
    emitter_storage.type    memory
    emitter_mem_buf_limit    120MB
[Filter]
    Name    lua
    Match    service.*
    script    /fluent-bit/config/systemd.lua
    call    add_time
    time_as_table    true
[Output]
    Name    null
    Match   *

This field is set by the clusterfluentbitconfigs.fluentbit.fluent.io.

kubectl get clusterfluentbitconfigs.fluentbit.fluent.io fluent-bit-config -oyaml
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-operator
    meta.helm.sh/release-namespace: fluent
  creationTimestamp: "2024-07-17T02:34:51Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit-config
  resourceVersion: "10179094"
  uid: bf63780e-f12f-411b-9ec0-e445f0cd6cad
spec:
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  multilineParserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  parserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    httpServer: true
    parsersFiles:
    - /fluent-bit/etc/parsers.conf
    - /fluent-bit/config/parsers_multiline.conf

Thank you let me try this

[meshpaul]

Many thanks for sticking with me on this!!!! But I not sure what is going on as I am unable to apply your suggestion even tho it's part of fluentbit.fluent.io_clusterfluentbitconfigs.yaml https://github.com/fluent/fluent-operator/blob/1d758ceab090b13a3a807292aeebd6b2fdef391e/charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusterfluentbitconfigs.yaml#L359

I was able to add logLevel and toggle various modes. I added parsersFile: /fluent-bit/config/parsers_multiline.conf worked

but as soon as I try to add parsersFiles:

I get the following error strict decoding error: unknown field "spec.service.parsersFiles"

Even the multi-line parser is applied as java_multiline I suspect that I need /fluent-bit/etc/parsers.conf to set as well. But I am unable because of the above mentioned error.

The fluent bit pods no longer crash The capture of java/spring-boot application stack trace behavior is still broken

Here is full ClusterFluentBitConfig

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  annotations:
    meta.helm.sh/release-name: fluent
    meta.helm.sh/release-namespace: fluent
  creationTimestamp: "2024-07-16T20:43:00Z"
  generation: 9
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit-config
  resourceVersion: "98028842"
  uid: c84eb69d-78c3-4add-891b-18fd9b6c5261
spec:
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  parserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    httpServer: true
    logLevel: info
    parsersFile: /fluent-bit/config/parsers_multiline.conf
    parsersFiles:
      - /fluent-bit/etc/parsers.conf
      - /fluent-bit/config/parsers_multiline.conf

Here is full error stack

# * patch: Invalid value: "{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"kind\":\"ClusterFluentBitConfig\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"fluent\",\"meta.helm.sh/release-namespace\":\"fluent\"},\"creationTimestamp\":\"2024-07-16T20:43:00Z\",\"generation\":9,\"labels\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"fluent-bit\"},\"managedFields\":[{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\".\":{},\"f:filterSelector\":{},\"f:inputSelector\":{},\"f:outputSelector\":{},\"f:parserSelector\":{},\"f:service\":{}}},\"manager\":\"terraform-provider-helm_v2.14.0_x5\",\"operation\":\"Update\",\"time\":\"2024-07-16T20:43:00Z\"},{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:service\":{\"f:httpServer\":{},\"f:logLevel\":{},\"f:parsersFile\":{}}}},\"manager\":\"kubectl-edit\",\"operation\":\"Update\",\"time\":\"2024-07-19T20:22:03Z\"}],\"name\":\"fluent-bit-config\",\"resourceVersion\":\"98028842\",\"uid\":\"c84eb69d-78c3-4add-891b-18fd9b6c5261\"},\"spec\":{\"filterSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"inputSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"outputSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"parserSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"service\":{\"httpServer\":true,\"logLevel\":\"info\",\"parsersFile\":\"/fluent-bit/config/parsers_multiline.conf\",\"parsersFiles\":[\"/fluent-bit/etc/parsers.conf\",\"/fluent-bit/config/parsers_multiline.conf\"]}}}": strict decoding error: unknown field "spec.service.parsersFiles"
#

Here is snippet of changed values.yaml for fluent-operator tail section

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log, /var/log/containers/*_keycloak_*.log, /var/log/containers/*_database_*.log "
      skipLongLines: true
      readFromHead: false
      storageType: memory
      pauseOnChunksOverlimit: "off"
      multilineParser: "java-multiline"

This no longer crash fluent-bit pods as part of values.yaml

  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline

Let me know if there is another way to work around this.

[benjaminhuo]

You might need to take a look at https://github.com/fluent/fluent-operator/pull/1138

[wanjunlei]

Many thanks for sticking with me on this!!!! But I not sure what is going on as I am unable to apply your suggestion even tho it's part of fluentbit.fluent.io_clusterfluentbitconfigs.yaml

https://github.com/fluent/fluent-operator/blob/1d758ceab090b13a3a807292aeebd6b2fdef391e/charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusterfluentbitconfigs.yaml#L359

I was able to add logLevel and toggle various modes. I added parsersFile: /fluent-bit/config/parsers_multiline.conf worked

but as soon as I try to add parsersFiles:

I get the following error strict decoding error: unknown field "spec.service.parsersFiles"

Even the multi-line parser is applied as java_multiline I suspect that I need /fluent-bit/etc/parsers.conf to set as well. But I am unable because of the above mentioned error.

The fluent bit pods no longer crash The capture of java/spring-boot application stack trace behavior is still broken

Here is full ClusterFluentBitConfig

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  annotations:
    meta.helm.sh/release-name: fluent
    meta.helm.sh/release-namespace: fluent
  creationTimestamp: "2024-07-16T20:43:00Z"
  generation: 9
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit-config
  resourceVersion: "98028842"
  uid: c84eb69d-78c3-4add-891b-18fd9b6c5261
spec:
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  parserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    httpServer: true
    logLevel: info
    parsersFile: /fluent-bit/config/parsers_multiline.conf
    parsersFiles:
      - /fluent-bit/etc/parsers.conf
      - /fluent-bit/config/parsers_multiline.conf

Here is full error stack

# * patch: Invalid value: "{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"kind\":\"ClusterFluentBitConfig\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"fluent\",\"meta.helm.sh/release-namespace\":\"fluent\"},\"creationTimestamp\":\"2024-07-16T20:43:00Z\",\"generation\":9,\"labels\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"fluent-bit\"},\"managedFields\":[{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\".\":{},\"f:filterSelector\":{},\"f:inputSelector\":{},\"f:outputSelector\":{},\"f:parserSelector\":{},\"f:service\":{}}},\"manager\":\"terraform-provider-helm_v2.14.0_x5\",\"operation\":\"Update\",\"time\":\"2024-07-16T20:43:00Z\"},{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:service\":{\"f:httpServer\":{},\"f:logLevel\":{},\"f:parsersFile\":{}}}},\"manager\":\"kubectl-edit\",\"operation\":\"Update\",\"time\":\"2024-07-19T20:22:03Z\"}],\"name\":\"fluent-bit-config\",\"resourceVersion\":\"98028842\",\"uid\":\"c84eb69d-78c3-4add-891b-18fd9b6c5261\"},\"spec\":{\"filterSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"inputSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"outputSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"parserSelector\":{\"matchLabels\":{\"fluentbit.fluent.io/enabled\":\"true\"}},\"service\":{\"httpServer\":true,\"logLevel\":\"info\",\"parsersFile\":\"/fluent-bit/config/parsers_multiline.conf\",\"parsersFiles\":[\"/fluent-bit/etc/parsers.conf\",\"/fluent-bit/config/parsers_multiline.conf\"]}}}": strict decoding error: unknown field "spec.service.parsersFiles"
#

Here is snippet of changed values.yaml for fluent-operator tail section

  input:
    tail:
      enable: true
      refreshIntervalSeconds: 10
      memBufLimit: 100MB
      bufferMaxSize: ""
      path: "/var/log/containers/*_dev_*.log, /var/log/containers/*_stg_*.log, /var/log/containers/*_keycloak_*.log, /var/log/containers/*_database_*.log "
      skipLongLines: true
      readFromHead: false
      storageType: memory
      pauseOnChunksOverlimit: "off"
      multilineParser: "java-multiline"

This no longer crash fluent-bit pods as part of values.yaml

  filter:
    multiline:
      enable: true
      keyContent: log
      # emitterMemBufLimit 120 (MB)
      emitterMemBufLimit: 120 
      parsers:
        - go
        - python
        - java
        - cri
        #  use custom multiline parser need set .Values.parsers.javaMultiline.enable = true
        - java-multiline

Let me know if there is another way to work around this.

Try to update the crds manually.

[meshpaul]

OK thank you... I know understand why this happened!!! Helm does not mange already installed CRDs. So when doing an update one ether expected to update these manually or in my case... I have re-installed the version. But old CRDs were not removed/updated... As result this caused above problem.

I am just going to wipe this clean including CRDs

[meshpaul]

Ok after full clean re-install

the fluent bit pods start as expected...

I now see recommended changes in ClusterFluentBitConfig

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  annotations:
    meta.helm.sh/release-name: fluent
    meta.helm.sh/release-namespace: fluent
  creationTimestamp: "2024-07-22T19:16:32Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
  name: fluent-bit-config
  resourceVersion: "99310781"
  uid: a6441bae-bfdc-47d6-9948-058396a93dd2
spec:
  filterSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  multilineParserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  parserSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    httpServer: true
    parsersFiles:
    - /fluent-bit/etc/parsers.conf
    - /fluent-bit/config/parsers_multiline.conf

[meshpaul]

Testing Multiline

[meshpaul]

OK I am looking a supplied recommended option as per below... the option are set as per below and I can't get java/spring boot stack traces to be part of the same message.

https://github.com/fluent/fluent-operator/pull/1138/files

[meshpaul]

Let me know if I should keep this issue open or close this (since my original installation of operator 3.0.0 was referencing old CRDs related to 2.X) and open a new one so we can start fresh?

[cw-Guo]

@meshpaul Please create another issue to track the functionality of Multiline parser. Thanks.

[benjaminhuo]

The crd upgrade guide is in the readme