Skip_Empty_Lines in Tail doesn't work

Albelt commented 2 years ago

version: 1.8 deployment: kubernetes
image: fluent/fluent-bit:1.8 problem: Skip_Empty_Line seems doesn't work in input plugin Tail my config file:

[INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Skip_Empty_Lines  On
        Refresh_Interval  10
.......

[OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

record in elasticsearch:

  "_source": {
    "@timestamp": "2021-10-22T02:49:09.952Z",
    "log": "\n",
    "time": "2021-10-22T02:49:09.952519448Z",
    "kubernetes_pod_name": "accountservice-7766f4b7bf-nv9xf",
    "kubernetes_namespace_name": "backend",
    "kubernetes_host": "k8s-node2-238"
  }

the log field of some records is '\n', shouldn't it be skip by this configuration?

nokute78 commented 2 years ago

Below config works. Could you share log ?

a.conf:

[SERVICE]
    Parsers_File parsers.conf

[INPUT]
    Name tail
    Path a.log
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Skip_Empty_Lines On
    Refresh_Interval 10
    Read_From_Head on

[OUTPUT]
    Name stdout

a.log:

aaa

bbb

ccc

$ ../bin/fluent-bit -c a.conf 
Fluent Bit v1.8.8
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/23 07:32:18] [ info] [engine] started (pid=4080)
[2021/10/23 07:32:18] [ info] [storage] version=1.1.4, initializing...
[2021/10/23 07:32:18] [ info] [storage] in-memory
[2021/10/23 07:32:18] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/23 07:32:18] [ info] [cmetrics] version=0.2.2
[2021/10/23 07:32:18] [ info] [sp] stream processor started
[2021/10/23 07:32:18] [ info] [input:tail:tail.0] inotify_fs_add(): inode=561437 watch_fd=1 name=a.log
[0] tail.0: [1634941938.559007082, {"log"=>"aaa"}]
[1] tail.0: [1634941938.559025649, {"log"=>"bbb"}]
[2] tail.0: [1634941938.559025870, {"log"=>"ccc"}]
^C[2021/10/23 07:32:23] [engine] caught signal (SIGINT)
[2021/10/23 07:32:23] [ info] [input] pausing tail.0
[2021/10/23 07:32:23] [ warn] [engine] service will stop in 5 seconds

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 5 days with no activity.

avizen-j commented 9 months ago

Sharing in case someone else is trying to figure out why that might be happening for you - https://github.com/fluent/fluent-bit/issues/8313

fluent / fluent-bit

Skip_Empty_Lines in Tail doesn't work #4215