Fails if parsed log does not contain syslog headers

[n473-]

If the incoming message is devoid of the %{SYSLOGBASE}, or has a slightly different format, the full message is still mutated to syslog_message, leaving "message" empty. This will cause problems if grok patterns are matched to "message".

For example:

Filter:

    grok {
        patterns_dir => "/etc/logstash/grok/squid.pattern"
        match => { message => "%{SQUID}" }
        add_tag => "squid"
    }

Output:

{
"@version" => "1",
"@timestamp" => "2014-05-15T13:10:09.000Z",
"type" => "syslog",
"host" => "x.x.x.x",
"priority" => 133,
 "timestamp" => "May 15 15:10:09",
 "logsource" => "squid-host",
 "program" => "squid-access-log",
 "severity" => 5,
 "facility" => 16,
 "facility_label" => "local0",
 "severity_label" => "Notice",
 "tags" => [
        [0] "_grokparsefailure"
 ],
 "syslog_severity_code" => 5,
 "syslog_facility_code" => 1,
 "syslog_facility" => "user-level",
 "syslog_severity" => "notice",
 "syslog_message" => "1400159402.631      0 172.17.14.67 TCP_MEM_HIT/301 820 GET http://google.co.za/ - NONE/- text/html"
}

Note how the [message] tag is empty or non-existent. Thus when grok attempts to match [message], it results in a _grokparsefailure.

[n473-]

This can be fixed with the use of an additional grok:

    # Pull out the Syslog Priority
    grok {
    match => { "message" => "<%{INT:syslog_pri}>" }
    }
   # Check if the message has syslog headers
    grok {
    match => { "message" => "%{SYSLOGBASE}" }
    add_tag => "syslogbase"
    }

    # Convert the syslog priority it to human readable
    syslog_pri { }

    # Move the Syslog Message into a new field named syslog_message (to capture the original)
    if ("syslogbase" in [tags]) {
      mutate {
          rename => ["message", "syslog_message"]
      }
       # Apply the SYSLOG GROK to the message to parse out the syslog headers, then place the left overs
       # in message where we expect it to be.
      grok {
          match => ["syslog_message", "%{SYSLOGBASE} %{GREEDYDATA:message}"]
      }
    }

[coolacid]

Which example is this? I'll clarify the documentation as this is Specifically related to remote syslog, not watching a file.

[n473-]

This is inclusive of remote syslog messages that for whatever reason do not arrive at the logstash rsyslog input with additional syslog headers. I'm not suggesting you modify the source, it's rather just an FYI in case anyone else has the same issues as I did :)