search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.82k stars 1.58k forks source link

multiline parser could not load #6547

Closed vijaykumar219 closed 1 year ago

vijaykumar219 commented 1 year ago

Bug Report

Describe the bug Hi I need to configure multiline parsing for my k8s env application. "Multiline could not load" error getting

Fluent Bit v2.0.6

[2022/12/14 04:34:02] [error] [lib] backend failed [2022/12/14 04:34:02] [ info] [fluent bit] version=2.0.6, commit=211f841fe2, pid=1 [2022/12/14 04:34:02] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2022/12/14 04:34:02] [ info] [cmetrics] version=0.5.7 [2022/12/14 04:34:02] [ info] [ctraces ] version=0.2.5 [2022/12/14 04:34:02] [ info] [input:tail:tail.0] initializing [2022/12/14 04:34:02] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only) [2022/12/14 04:34:02] [error] [multiline] parser 'multiline-regex-test' not registered [2022/12/14 04:34:02] [error] [input:tail:tail.0] could not load multiline parsers [2022/12/14 04:34:02] [error] failed initialize input tail.0 [2022/12/14 04:34:02] [error] [engine] input initialization failed

To Reproduce


- Steps to reproduce the problem: Configuration of environment below

**Expected behavior**
<!--- A clear and concise description of what you expected to happen. -->
A single line output in grafana from multiline to single line.
**Screenshots**
<!--- If applicable, add screenshots to help explain your problem. -->
![image](https://user-images.githubusercontent.com/32031066/207510810-1a953e59-7632-4150-a1f7-2875ebf75ae4.png)

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: helm chart  (fluent/fluent-bit               0.21.5          2.0.6)
* Configuration:
* apiVersion: v1
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers_multiline.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Tag kube.*
        read_from_head        true
        multiline.parser docker, cri, multiline-regex-test
        Buffer_Chunk_Size 32k
        Buffer_Max_Size 32k
        Mem_Buf_Limit 50MB
        Skip_Long_Lines off

    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_URL https://kubernetes.default.svc:443
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name loki
        Match kube.*
        Host loki-loki-distributed-distributor.loki-grafana.svc.cluster.local
        tenant_id ""
        labels job=fluentbit
        remove_keys kubernetes,stream
        line_format json
        auto_kubernetes_labels off
  parsers_multiline.conf: |
    [MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    flush_timeout 1000
    # rules   |   state name   | regex pattern                   | next state name
    # --------|----------------|--------------------------------------------------
    rule         "start_state"   "/(([\d-]+ [\d:,]+))(.*)/"  "cont"
    rule         "cont"          "/^\s+at.*/"                     "cont"

* Environment name and version (e.g. Kubernetes? What version?): kubernetes v1.19.15
* Server type and version:
* Operating System and version:
* Filters and plugins: fluentbit ->loki(2.6) ->grafana(9.3.1)

**Additional context**
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
I tried to set multiline.parser multiline-regex-test and it provides an error "**multiline regex not registered**"
RicardoAAD commented 1 year ago

Hello @vijaykumar219, Your config looks good, but the reason why you are getting this error is due to the missing comments in your multiline parser

it looks like this:

[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000

rules | state name | regex pattern | next state name
--------|----------------|--------------------------------------------------
rule "start_state" "/(([\d-]+ [\d:,]+))(.)/" "cont"
rule "cont" "/^\s+at./" "cont"

But should be

        name multiline-regex-test
        type regex
        flush_timeout 1000
        # rules    | state name        | regex pattern                | next state name
        #--------|----------------|------------------------|--------------------------
        rule       "start_state"   "/(([\d-]+ [\d:,]+))(.)/"      "cont"
        rule       "cont"          "/^\s+at./"                    "cont"

These two lines are not needed; they were only added to simplify the definition of rules Rules Definition

rules | state name | regex pattern | next state name
--------|----------------|--------------------------------------------------

Please let us know if, after commenting or removing these lines, the multiline parser works for you.

Regards, Ricardo

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 5 days with no activity.