search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.8k stars 1.58k forks source link

How to use few fields in grep filter? #594

Closed bat9r closed 6 years ago

bat9r commented 6 years ago

I`m using fluent-bit:0.13.0, i want grep "warns", "errors" and etc. from multiple fields using grep filter. How i can do it? This is part of my config.

filter-kubernetes.conf: |
    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc.cluster.local:443
        Merge_JSON_Log On
        tls.verify     Off
        tls.debug      4
    [FILTER]
        Name           grep
        Match          kube.*
        Regex          log|msg (?i)warn|error|exception|critical|warning|err

How to grep from "log" and "msg" field if they comply regex?

edsiper commented 6 years ago

@bat9r It depends on the format of your incoming logs, I would put the grep filter before kubernetes filter, so you let pass the only ones you care about.

bat9r commented 6 years ago

@edsiper Thank you for answer. Soo, i can't "care about" for two fields? Or i could do it from other side (an application which aggregate logs)?

michiel commented 6 years ago

A workaround might be (on 0.14 - currently unreleased) to apply the modify filter twice with conditions. This allows you to create an OR condition where a record is tagged with k/v keep:true if it matches either condition. You can then apply the grep filter to keep or drop the record based on that k/v.

   [FILTER]
       Name           modify
       Match          kube.*
       Condition      Key_value_matches log REGEX_TO_MATCH
       Set            keep true

   [FILTER]
       Name           modify
       Match          kube.*
       Condition      Key_value_matches msg REGEX_TO_MATCH
       Set            keep true

   [FILTER]
       Name           grep
       Match          kube.*
       Regex          keep true
bat9r commented 6 years ago

@michiel Thank you, great idea :)

viveksinghggits commented 5 years ago

I`m using fluent-bit:0.13.0, i want grep "warns", "errors" and etc. from multiple fields using grep filter. How i can do it? This is part of my config.

filter-kubernetes.conf: |
   [FILTER]
       Name           kubernetes
       Match          kube.*
       Kube_URL       https://kubernetes.default.svc.cluster.local:443
       Merge_JSON_Log On
       tls.verify     Off
       tls.debug      4
   [FILTER]
       Name           grep
       Match          kube.*
       Regex          log|msg (?i)warn|error|exception|critical|warning|err

How to grep from "log" and "msg" field if they comply regex?

Hi @bat9r , does line Regex log|msg (?i)warn|error|exception|critical|warning|err mean that I want to filter the message where either log or msg field is among warn|error|exception|critical|warning|err. If yes, I wanted to filter the logs where container_name is either nifi or redis I used below config, which doesnt seem to work Regex CONTAINER_NAME nifi | redis any suggestion ?