Multiple filtering logs using regular expressions

[wobeng]

Hello,

I have ecs config that pushes logs to awsfirelens. Currently, all the logs go to cloudwatch log group A but I want to

1. look for a certain pattern in the logs
2. exclude it from going to cloudwatch log group A
3. send it to another cloudwatch log group B

Per Filtering logs using regular expressions, you can filter logs using regular expressions; however, I don't think it supports multiple log configurations

I have the config below; however, everything is going to cloudwatch log group A and nothing in B. I do not see any error in the logs for this aws-for-fluent-bit image. I have tested the regex against my logs and it works fine. see this

[SERVICE]
    Flush     1
    Log_Level debug

[FILTER]
    Name  rewrite_tag
    Match *
    Rule  $log "^.+— timestream — INFO —.+$"  timestream false
    Emitter_Name  re_emitted

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name /aws/ecs/app-${ENVIRONMENT}
    log_stream_prefix ecs/
    auto_create_group On
    log_key log

[OUTPUT]
    Name cloudwatch_logs
    Match   timestream*
    region us-east-1
    log_group_name /aws/ecs/timestream-${ENVIRONMENT}
    log_stream_prefix ecs/
    auto_create_group On
    log_key log

cloudwatch log group A = /aws/ecs/app-${ENVIRONMENT} cloudwatch log group B = /aws/ecs/timestream-${ENVIRONMENT}

[zhonghui12]

Hi,

so this doc is for filtering logs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-filtering-logs.html. And the rewrite_tag filter you posted above is for rewriting tags: https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag. It will not check logs based on the rule you defined.

[wobeng]

@zhonghui12, thanks for your response. The link you pointed me to works if there is one destination, but I have two. Also, can you clarify

the rewrite_tag filter you posted above is for rewriting tags; it will not check logs based on the rule you defined.

I thought the rewrite_tag is to rewrite the tag based on a regex that matches the log?

[zhonghui12]

I see. I think this blog can help you achieve your goal: https://aws.amazon.com/blogs/opensource/splitting-application-logs-multiple-streams-fluent/

[wobeng]

any thoughts @PettitWesley :)

[matthewfala]

Hi @wobeng, I suspect the error is one of the following:

1. The regex is not identifying the logs you want to retag.
2. The Match is picking up all the logs so timestream gets none.
3. The expected log key is not arriving at fluent bit.

To diagnose 1., would you please open: https://regexr.com/ and type into the top regex box your regex:

^.+— timestream — INFO —.+$

And in the box below write a log you hope will make it to the timestream log group. There should be a regex match. If the regex does not match, then 1. is your problem.

If the regex matches, then 1. is not the problem: To diagnose 2. instead of Match * for your second log group, could you please replace that with Match abc which shouldn't match any of your tags. Please check to see if you are receiving logs to your timestream group. If you see logs arriving at your timestream group (B) then 2. is your problem.

If you are not seeing logs arrive at timestream group (B) then please try the following to diagnose 3: Delete all the outputs from your config file. Replace with the output:

[OUTPUT]
    Name stdout
    Match *

You should see logs in your standard out console in the format: [0] tag: [timestamp, data]. Confirm that you are actually receiving the log key you are expecting to activate the regex tag replacement rule.

In any case, would you please add a sample log that you expect to arrive in your group a, and a sample log that you expect to arrive from group b from your stdout logs? Let me know what you find.

[matthewfala]

A teammate recommends using rubular.com to test your regex as it is closer to fluent bit's regex system.