search

logstash-plugins / logstash-filter-dissect

Extract structured fields from an unstructured line
Apache License 2.0
13 stars 22 forks source link

option to ignore missing fields at the end of a log #62

Open mmguero opened 6 years ago

mmguero commented 6 years ago

This is an enhancement request.

The documentation says:

When dissecting a string from left to right, text is captured upto the first delimiter - this captured text is stored in the first field. This is repeated for each field/# delimiter pair thereafter until the last delimiter is reached, then the remaining text is stored in the last field.

However, when you have the opposite problem (eg., if you have missing fields instead of the correct number of fields or missing fields) you get a "Dissector mapping, pattern not found" error.

This makes it difficult to use one set of rules to parse logs with a format that evolves over time, adding new fields to the end.

It would be very convenient if there were an option to just set the missing fields to nil, or not add them to the event at all.

guyboertje commented 6 years ago

We will be adding this [1] fairly soon. Its coded up but needs some tidying. [1] https://github.com/logstash-plugins/logstash-filter-dissect/issues/56 You may find it will help to define a number of patterns that will get tried until success.

mmguero commented 6 years ago

I see, thanks for the comment. Yes, I think that would allow me to do what I'm attempting. I'll look forward to the next release.

mmguero commented 5 years ago

Just curious, do you have any thoughts on when #56 will be ready for release? Just trying to gauge what "fairly soon" means for whether or not I'll be able to use that enhancement in a project I'm working on.