logstash-plugins / logstash-filter-dissect

Extract structured fields from an unstructured line
Apache License 2.0
13 stars 22 forks source link

Entire message parsed in last field #68

Closed Baplar closed 5 years ago

Baplar commented 5 years ago

When trying to dissect a '#'-separated message with a bunch of empty fields, the result I obtain populates every field with an empty string, except for the last one which receives the entire message.

What makes it weirder is that this dissect is performed right after a first one which does exactly what I expect from it.

This second parsing phase uses two consecutive delimitors at one point, but adding a skip field between them did not solve the problem.

Filter config: (some fields skipped for concision and confidentiality)

filter {
    dissect {
        id => "message_parse_1"
        mapping => {
            "message" => "#%{timestamp}#%{sequence_number}#%{}#%{}#%{}#%{}#%{}#%{}#%{criticality_level}#%{application_identifier}#%{message_remaining}"
        }
    }
    if [app_id] == "APP1" {
        dissect {
            id => "message_parse_2"
            mapping => {
                "message_remaining" => "%{}#%{}#%{}#%{}##%{app_tag}#%{}#%{event_code}#%{}#%{description}"
            }
        }
    }
}

Sample data:

#20190219-060046#5553#######INFO#APP1######TAG1##42##|metric_1|0|NOMINAL||

Expected result: (logstash-specific and empty fields truncated)

{
    "message": "#20190219-060046#5553#######INFO#APP1######TAG1##42##|metric_1|0|NOMINAL||",
    "timestamp": "20190219-060046",
    "sequence_number": "5553"
    "criticality_level": "INFO"
    "app_id": "APP1",
    "message_remaining": "#####TAG1##42##|metric_1|0|NOMINAL||",
    "app_tag": "TAG1"
    "event_code": "42"
    "description": "|metric_1|0|NOMINAL||"
}

Obtained result: (truncated)

{
    "message": "#20190219-060046#5553#######INFO#APP1######TAG1##42##|metric_1|0|NOMINAL||",
    # [...]
    "message_remaining": "#####TAG1##42##|metric_1|0|NOMINAL||",
    "description": "#####TAG1##42##|metric_1|0|NOMINAL||"
}

I know that the 2nd dissect is being processed because it creates all the corresponding fields, but it seems to silently fail during parsing and just gather everything in the last field. Did I just do something wrong, or is that unexpected behavior?

guyboertje commented 5 years ago

Since this is a general usage question and not a reproducible bug report or a clear feature request, I am closing this issue. Someone will help on the forum.

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.