MULTILINE_PARSER is not detected after chart installation

asenyaev commented 6 months ago

Bug Report

Hello! I have faced an issue when I try to add [MULTILINE_PARSER] block in customParsers field of fluent-bit chart following the official documentation and defining this parser in one of filters in [FILTER] block, which throws an error:

To Reproduce

Define following configs to the default values in fluent-bit helm chart

config:
  filters: |
    [FILTER]
            Name           multiline
            Match          *
            multiline.parser multiline-regex

  customParsers: |
    [MULTILINE_PARSER]
        name          multiline-regex
        type          regex
        flush_timeout 1000

        # rules |   state name  | regex pattern                    | next state
        # ------|---------------|----------------------------------|-----------
        rule      "start_state"   "/(Mar \d+ \d+\:\d+\:\d+)(.*)/"    "cont"
        rule      "cont"          "/^\s+at.*/"                       "cont"

Apply it

Pod is in CrashLoopBackOff state with the following error:

[error] [multiline] parser 'multiline-regex' not registered

Expected behavior I would like to use custom multiline parser to parse my logs properly. Where am I wrong in a configuration or multiline is broken?

Your Environment

Version used: v2.2.2
Environment name and version (e.g. Kubernetes? What version?): k8s 1.28

patrick-stephens commented 6 months ago

I would raise this on the helm-charts repo really.

Also, check for other errors in the log - having the full pod log would be better. It may be complaining about something else that just happens to manifest as a parser not registered error.

patrick-stephens commented 6 months ago

Also, I tried to reproduce this locally without any issues:

cat values.yaml

config:
  filters: |
    [FILTER]
            Name           multiline
            Match          *
            multiline.parser multiline-regex

  customParsers: |
    [MULTILINE_PARSER]
        name          multiline-regex
        type          regex
        flush_timeout 1000

        # rules |   state name  | regex pattern                    | next state
        # ------|---------------|----------------------------------|-----------
        rule      "start_state"   "/(Mar \d+ \d+\:\d+\:\d+)(.*)/"    "cont"
        rule      "cont"          "/^\s+at.*/"                       "cont"

  outputs: |
     [OUTPUT]
             name stdout
             match *

Installed:

kind create cluster
helm repo update
helm upgrade --install --values=$PWD/values.yaml --wait fb fluent/fluent-bit
kubectl logs $(kubectl get pods --namespace default -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fb" -o jsonpath="{.items[0].metadata.name}")
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/03/12 14:47:05] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1
[2024/03/12 14:47:05] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/12 14:47:05] [ info] [cmetrics] version=0.6.6
[2024/03/12 14:47:05] [ info] [ctraces ] version=0.4.0
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] initializing
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] multiline core started
[2024/03/12 14:47:05] [ info] [input:systemd:systemd.1] initializing
[2024/03/12 14:47:05] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2024/03/12 14:47:05] [ info] [filter:multiline:multiline.0] created emitter: emitter_for_multiline.0
[2024/03/12 14:47:05] [ info] [input:emitter:emitter_for_multiline.0] initializing
[2024/03/12 14:47:05] [ info] [input:emitter:emitter_for_multiline.0] storage_strategy='memory' (memory only)
[2024/03/12 14:47:05] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/12 14:47:05] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/03/12 14:47:05] [ info] [sp] stream processor started
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=929451 watch_fd=1 name=/var/log/containers/coredns-76f75df574-cqxn4_kube-system_coredns-810f7067e99ce48c1e329fab752ab19721512bcb18320f48393f42201714409a.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=929495 watch_fd=2 name=/var/log/containers/coredns-76f75df574-t5q2t_kube-system_coredns-4e0200afd6494821833057fc0759f223542caf3b0abf04b7fcbf29638b45dd68.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=920451 watch_fd=3 name=/var/log/containers/etcd-kind-control-plane_kube-system_etcd-db7ee771c5c9a57587e4f7269401fe00d35a56c224949660460f72fefa5132e7.log
[2024/03/12 14:47:05] [ info] [filter:multiline:multiline.0] created new multiline stream for tail.0_kube.var.log.containers.fb-fluent-bit-2j6mv_default_fluent-bit-f1bb0179e36f56e4cba237f931d63c21464033be042424cf0d3be35ebc49134d.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=926729 watch_fd=4 name=/var/log/containers/kindnet-qqkql_kube-system_kindnet-cni-4e3be1b5d47db121c1d95a1dd77cd88f9703443a0ddaed099256407ba92f0b2b.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=918920 watch_fd=5 name=/var/log/containers/kube-apiserver-kind-control-plane_kube-system_kube-apiserver-0c71430fef214f21e8b106539edecced4e4f5cb4673c24057dfdefee1925deea.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=919160 watch_fd=6 name=/var/log/containers/kube-controller-manager-kind-control-plane_kube-system_kube-controller-manager-a7440f19cc63a24bf26471994aecb48ded1f31224cd2a6ca4fd26339229c6406.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=925198 watch_fd=7 name=/var/log/containers/kube-proxy-4tpjc_kube-system_kube-proxy-0e5828143c544565f307dd33ea20dd8f7034b60e836024529d08636ccb17a955.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=919481 watch_fd=8 name=/var/log/containers/kube-scheduler-kind-control-plane_kube-system_kube-scheduler-e341ad0b149f7dc49ecc0e54fbb4a8029871048e2fe6bd9fd238c25126ef41d8.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=929554 watch_fd=9 name=/var/log/containers/local-path-provisioner-7577fdbbfb-k7q7n_local-path-storage_local-path-provisioner-3e3598928bec0db880468d76b9db931478b5fbb5ec9b3ad2288703382c86d9bf.log
[2024/03/12 14:47:05] [ info] [input:tail:tail.0] inotify_fs_add(): inode=932204 watch_fd=10 name=/var/log/containers/fb-fluent-bit-2j6mv_default_fluent-bit-f1bb0179e36f56e4cba237f931d63c21464033be042424cf0d3be35ebc49134d.log
[0] kube.var.log.containers.fb-fluent-bit-2j6mv_default_fluent-bit-f1bb0179e36f56e4cba237f931d63c21464033be042424cf0d3be35ebc49134d.log: [[1710254825.356320422, {}], {"time"=>"2024-03-12T14:47:05.356320422Z", "stream"=>"stderr", "_p"=>"F", "log"=>"[2024/03/12 14:47:05] [ info] [input:systemd:systemd.1] initializing"}]

asenyaev commented 6 months ago

@patrick-stephens yes, you are right, it works properly. Thank you for the investigation!

In my case the wrong regex in a another (commented) multiline parser was wrong, what breaks the proper one.

Closing the issue.

fluent / helm-charts

MULTILINE_PARSER is not detected after chart installation #469

Bug Report