It takes several minutes to establish a connection

[MPeli]

Bug Report

Describe the bug I have 120 files (~500MB) in a folder. When fluent bit is started, it takes several minutes to establish a connection. Once a connection is established, several log lines (~387) are sent to ElasticSearch. Fluent bit never sends more logs.

To Reproduce

• Build fluent bit v1.7.6 on Windows or download https://fluentbit.io/releases/1.7/td-agent-bit-1.7.6-win64.exe. Prepare 120 log files. They should have roughly 500Mb in total. Clear backlog and delete db file. Start fluent bit and wait, check a log file.

Expected behavior Fluent bit should establish a connection within several seconds.

Your Environment

• Version used: v1.7.6
• Server type and version: Windows 10
• Filters and plugins: tail, es

Additional context We have several servers, each server has hundreds of files which we would like to monitor and send to AWS ElasticSearch.

[Fluent Bit]
    Version             1.7.6
    Built Flags          FLB_HAVE_PARSER FLB_HAVE_RECORD_ACCESSOR FLB_HAVE_STREAM_PROCESSOR JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_AWS FLB_HAVE_SIGNV4 FLB_HAVE_SQLDB FLB_HAVE_TRACE FLB_HAVE_TIMESPEC_GET FLB_HAVE_PROXY_GO FLB_HAVE_REGEX FLB_HAVE_UTF8_ENCODER FLB_HAVE_LUAJIT

[Operating System]
    Name                Windows
    Version             6.2
    Build               9200

[Hardware]
    Architecture        x64 (AMD or Intel)
    Processors          12

[Built Plugins]
    Inputs              emitter tail dummy statsd storage_backlog stream_processor winlog tcp lib forward random
    Filters             alter_size aws record_modifier throttle throttle_size kubernetes modify nest parser expect grep rewrite_tag lua stdout geoip2
    Outputs             azure azure_blob counter datadog es file forward http influxdb logdna loki nrlogs null slack splunk stackdriver stdout syslog tcp flowcounter gelf websocket cloudwatch_logs kinesis_streams s3

[SERVER] Runtime configuration
    Flush               5.000000
    Daemon              On
    Log_Level           Trace

[INPUT] Instance
    Name                tail.0 (tail, id=0)
    Flags
    Threaded            No
    Tag                 <appname>-<pid>
    Mem_Buf_Limit       95.4M
    Path                D:\path\*_*.Xlog.log,D:\path\*_*.Xlog.x64.log
    Multiline           on
    Parser_Firstline    xlog_parser
    Path_Key            path
    Offset_Key          1
    DB                  fluent.db
    Read_from_Head      On
    Tag_Regex           ^.*\\(?<appname>.+)_(?<pid>[0-9]+).Xlog.*.log$
    Buffer_Chunk_Size   128k
    Buffer_Max_Size     256k
    Ignore_Older        10d
    Routes              es.0

[INPUT] Instance
    Name                storage_backlog.1 (storage_backlog, id=1)
    Flags
    Threaded            No
    Tag                 storage_backlog.1
    Routes              es.0

[FILTER] Instance
    Name                grep.0 (grep, id=0)
    Match               *
    Exclude             severity SPAM
    Exclude             severity DEBUG
    Exclude             severity METHOD
    Regex               tid .+
[FILTER] Instance
    Name                lua.1 (lua, id=1)
    Match               *
    script              parse_ts.lua
    call                cb_parse_ts
[FILTER] Instance
    Name                lua.2 (lua, id=2)
    Match               *
    script              append_tag.lua
    call                append_tag
[FILTER] Instance
    Name                lua.3 (lua, id=3)
    Match               *
    script              conv.lua
    call                conv
    Type_int_key        pid
[FILTER] Instance
    Name                record_modifier.4 (record_modifier, id=4)
    Match               *
    Record              hostname HOSTNAME_CHANGED

[OUTPUT] Instance
    Name                es.0 (es, id=0)
    Match               *
    TLS Active          Yes
    TLS.Verify          Off
    TLS.Ca_File         (not set)
    TLS.Crt_File        (not set)
    TLS.Key_File        (not set)
    TLS.Key_Passwd      (not set)
    Retry Limit         no limit
    Host.TCP_Port       443
    Host.Name           abcd.eu-west-1.es.amazonaws.com
    Index               fluent-bit
    Logstash_Format     true
    HTTP_User           aaa
    HTTP_Passwd         bbb
    Trace_Output        On
    Trace_Error         On
    Buffer_Size         False

fluent-bit.log

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.