NR-279356: Add new multiline parser config param to logging config

SivaKumarP127 commented 2 months ago

Description

Adds a new config parameter which is Multiline.Parser to the INPUT block of fluent-bit config which enables the user to parse multiline logs. And this can be enabled by configuring the multilineParser for the log file in the logging.yml like below.

logging.yml 

 -  name: java.log
    file: /var/log/javalogs.log
    multilineParser: java
    attributes:
      logtype: java_logs

fluent-bit.conf

[INPUT]
    Name tail
    Path /var/log/javalogs.log
    Buffer_Max_Size 128k
    Mem_Buf_Limit 16384k
    Skip_Long_Lines On
    Multiline.Parser java        
    Path_Key filePath
    Tag  java.log
    DB   /var/db/newrelic-infra/newrelic-integrations/logging/fb.db

Note: Fluent-Bit already has built-in multiline parsers which are java, go, python, docker and cri. And the above changes will only support fluent-bit built-in multiline parsers. FYI: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#built-in-multiline-parsers

rubenruizdegauna commented 2 months ago

@SivaKumarP127 can you sign the commits please? Thanks!

SivaKumarP127 commented 2 months ago

Hi @rubenruizdegauna. Could you please let me know when these changes will be released? Thank you!

rubenruizdegauna commented 2 months ago

Hi @SivaKumarP127 , the idea is to create a release with that change this week. So let's think that next week should be already released.

rubenruizdegauna commented 2 months ago

Hi @SivaKumarP127 , yesterday we did a release that contains this functionality

https://github.com/newrelic/infrastructure-agent/releases/tag/1.53.0

newrelic / infrastructure-agent

NR-279356: Add new multiline parser config param to logging config #1881

Description