search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

log-courier 2.x: Multiline codec: Multiple patterns definitions not working #297

Closed jfrome closed 8 years ago

jfrome commented 8 years ago

When using the following configuration, no data is sent to logstash:

"codecs": [
    {
        "name": "multiline",
        "max multiline bytes": 10485760,
        "patterns": [
            "!^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}",
            "!^(|@)[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{4}"
        ],
        "what": "previous"
    }
],

Individually both patterns work, but when configured together, no output is the result.

driskell commented 8 years ago

As long as lines match any pattern they are buffered. In your configuration whatever doesn't match the second pattern would actually match the first and vice versa. I think due to the negate.

Thinking of ways to do this I wonder if a full negate is required so you can say buffer lines that don't match any pattern. Maybe there's a need for negate in pattern AND overall!

driskell commented 8 years ago

I've thrown in a new match parameter which defaults to any. If you set this to all it'll require that all patterns match. In your case, it'll require that all patterns don't match (since you negated them) and then you'll get your desired behaviour.

Your scenario is a common one (I have a huge amount of Log Courier 1.8 with ugly big patterns which I added multiple patterns to solve, but I would've hit your same issue on some of them!)

jfrome commented 8 years ago

Thanks! In initial testing it appears to be working as designed.