Multiple regexes

[Pehesi97]

My fluentd config has two regexes, but only the first one will match.

This is my .conf file.

<source>
  @type http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

<source>
    @type tail
    path /Users/pedro/.pm2/logs/fv-back-out*.log
    pos_file /Users/pedro/.pm2/logs/fv-back-response.log.pos
    tag fv-back
    <parse>
        @type multi_format
        <pattern>
            format regexp
            expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>(.)*response(.)*)\] data: (?<method>[^ ]+) (?<endpoint>[^ ]+) (?<query>[^ ]+) (?<payload>[^ ]+) (?<statusCode>[^ ]+) \((?<responseTime>[^ ]+)\) (?<credentials>[^ ]+)/
            time_format %Y-%m-%d %H:%M:%S.%L
            types time:time,tags:array,statusCode:integer
        </pattern>
        <pattern>
            format regexp
            expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>((.)*error(.)*))\] data: \[(?<statusCode>.*)\] (?<type>.*): (?<message>.*)/
            time_format %Y-%m-%d %H-%M-%S.%L
                    types time:time,tags:array,statusCode:integer
        </pattern>
    </parse>
</source>

<match fv-back*>
    @type copy

    <store>
        @type elasticsearch
        host localhost
        port 9200
        logstash_format true
    </store>

    <store>
        @type s3
        aws_key_id AKIAI3BKNP4PWLJW5PMQ
        aws_sec_key hoWhK+KzAKOwKuPszBkS8099Nuu8WZKJKo516XUY
        s3_bucket softruck.pm2.logs
        s3_region us-east-1
        path /

        <buffer time>
            @type file
            path /var/log/td-agent/s3_buffer
            timekey 3600
            timekey wait 10m
            timekey_use_utc true
            chunk_limit_size 256m
        </buffer>
    </store>
</match>

<match **>
    @type null
</match>

And this 2018-08-08 16:45:05 -0300 [warn]: #0 pattern not match: "2018-08-08 16:45:04.869 -03:00, [log,api,error] data: [500] SequelizeDatabaseError: relation \"daily_courses\" does not exist" is my Fluentd log.

Rubular says my regex should pass. Is there anything I'm doing wrong?

[ZhangSIming-blyq]

same issue here. wait for an answer.

[ggpaue]

same issue here, can anyone help?

[rsilva-rs]

You could try this plugin: https://github.com/sesame/fluent-plugin-regexp_multi

[davelosert]

We are having the same issue - is there a chance this will get fixed?

[ankit1mg]

We are having the same issue - is there a chance this will get fixed?

+1

[asdfII]

We are having the same issue - is there a chance this will get fixed?

+1

[cm0s]

Same issue. As anyone found a solution? @rsilva-rs plugin doesn't seem to exist anymore.

[repeatedly]

I think this is not a bug. Issue author's problem is the mismatch between log body and time format.

Second expression matches 2018-08-08 16:45:04.869 -03:00, [log,api,error] data: [500] SequelizeDatabaseError: relation "daily_courses" does not exist but time field, 2018-08-08 16:45:04.869, is %Y-%m-%d %H:%M:%S.%L format, not %Y-%m-%d %H-%M-%S.%L. Put more <pattern> fixes this problem. If you have similar pattern not match log, check your expression/time_format combination for actual logs.