Use Case: liblognorm’s alternative[s] Parsing Syntax

[preyniers]

Please include this use-case, or one like it, as part of the liblognorm examples documentation. If there is a more eloquent manner to express this rule base, please let me know.

Overview

This example details a use-case for liblognorm’s alternative syntax. The use-case is applied to inconsistent preambles for syslog messages. Through the use of alternative syntax, users can create API-style message syntax type[s] which allow for a consistent and succinct parser interface.

How-To

Versioning

lognormalizer version: 2.0.6
liblognorm version: 2.0.6

Directory Structure

<project directory>/parsers
<project directory>/types
<project directory/logs

Input File

Formats

Raw Message ($rawmsg)

<date-rfc5424><whitespace><from host><whitespace><colon><whitespace><cisco-asa ID><colon><message>
<date-rfc5424><whitespace><from host><whitespace><cisco-asa ID><colon><message>

Message ($msg):

<whitespace><cisco-asa ID><colon><message>
<cisco-asa ID><colon><message>

Data

2019-02-28T17:56:24+00:00 1.1.1.1 : %ASA-6-106015: Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface
2019-02-28T17:56:24+00:00 1.1.1.1  %ASA-6-106015: Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface
 %ASA-6-106015: Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface
%ASA-6-106015: Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface

Type File

version=2
#===============================================================================
# File: types/type_cisco-asa-support.rulebase
#
# Description:
#   This rulebase is intended to provide common support type[s] for cisco-asa
#   syslog messages.
#
# Revision:
#   7:06 AM Saturday, March 9, 2019
#===============================================================================

#===============================================================================
# Message Prefix Components
#===============================================================================
type=@MSG_PREFIX_DATE:%[
    { "type" : "date-rfc5424",
      "name" : "timestamp"
    }
  ]%
type=@MSG_PREFIX_FROMHOST:%[
    { "type" : "string",
      "name": "from_host"
    }
  ]%
type=@MSG_PREFIX_VENDOR_ID:%[
    { "type" : "string",
      "matching.mode"       : "lazy",
      "matching.permitted"  :
        [ { "chars" : "%%" } ]
    },
    { "type" : "char-to",
      "name" : "product",
      "extradata" : "-"
    },
    { "type" : "literal",
      "text" : "-"
    },
    { "type" : "number",
      "name" : "log_level"
    },
    { "type" : "literal",
      "text" : "-"
    },
    { "type" : "number",
      "name" : "message_id"
    },
    { "type" : "literal",
      "text" : ":"
    }
  ]%
#===============================================================================
# Message Prefix Formats
#===============================================================================
type=@MSG_PREFIX_FMT_1:%[
    { "type" : "@MSG_PREFIX_DATE",
      "name" : "."
    },
    { "type" : "whitespace"
    },
    { "type" : "@MSG_PREFIX_FROMHOST",
      "name" : "."
    },
    { "type" : "literal",
      "text" : " : "
    },
    { "type" : "@MSG_PREFIX_VENDOR_ID",
      "name" : "."
    }
  ]%
type=@MSG_PREFIX_FMT_2:%[
    { "type" : "@MSG_PREFIX_DATE",
      "name" : "."
    },
    { "type" : "whitespace"
    },
    { "type" : "@MSG_PREFIX_FROMHOST",
      "name" : "."
    },
    { "type" : "whitespace"
    },
    { "type" : "@MSG_PREFIX_VENDOR_ID",
      "name" : "."
    }
  ]%
type=@MSG_PREFIX_FMT_3:%[
    { "type" : "whitespace"
    },
    { "type" : "@MSG_PREFIX_VENDOR_ID",
      "name" : "."
    }
  ]%
type=@MSG_PREFIX_FMT_4:%[
    { "type" : "@MSG_PREFIX_VENDOR_ID",
      "name" : "."
    }
  ]%
#===============================================================================
# Alternative Message Prefix
#===============================================================================
type=@MSG_PREFIX:%{
  "type" : "alternative", "parser" :
    [ { "type" : "@MSG_PREFIX_FMT_1", "name" : "." },
      { "type" : "@MSG_PREFIX_FMT_2", "name" : "." },
      { "type" : "@MSG_PREFIX_FMT_3", "name" : "." },
      { "type" : "@MSG_PREFIX_FMT_4", "name" : "." }
    ]
  }%

#===============================================================================
#                                   END OF FILE
#===============================================================================

Parser File

version=2
#===============================================================================
# File: parsers/parser_cisco-asa.rulebase
#
# Description:
#   This rulebase is intended to provide common support parser[s] for cisco-asa
#   syslog messages.
#
# Revision:
#   7:18 AM Saturday, March 9, 2019
#===============================================================================

#===============================================================================
# Includes
# Set the <project directory> tag to an appropriate value.
#===============================================================================
include=<project directory>/types/type_cisco-asa-support.rulebase

prefix=%[{ "type" : "@MSG_PREFIX", "name" : "." }]%

#===============================================================================
# Rules
#===============================================================================
rule=cisco,asa,example,alternative:%[
  { "type" : "rest",    "name" : "_unparsed_data" }
]%

#===============================================================================
#                                   END OF FILE
#===============================================================================

Output

cat ./logs/t_test.log | lognormalizer -THU -r ./parsers/parser_cisco-asa.rulebase  | jq
4 records processed, 4 parsed, 0 unparsed
{
  "_unparsed_data": " Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface",
  "message_id": "106015",
  "log_level": "6",
  "product": "ASA",
  "from_host": "1.1.1.1",
  "timestamp": "2019-02-28T17:56:24+00:00",
  "event.tags": [
    "cisco",
    "asa",
    "example",
    "alternative"
  ]
}
{
  "_unparsed_data": " Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface",
  "message_id": "106015",
  "log_level": "6",
  "product": "ASA",
  "from_host": "1.1.1.1",
  "timestamp": "2019-02-28T17:56:24+00:00",
  "event.tags": [
    "cisco",
    "asa",
    "example",
    "alternative"
  ]
}
{
  "_unparsed_data": " Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface",
  "message_id": "106015",
  "log_level": "6",
  "product": "ASA",
  "event.tags": [
    "cisco",
    "asa",
    "example",
    "alternative"
  ]
}
{
  "_unparsed_data": " Deny TCP (no connection) from 2.2.2.2/49162 to 1.1.1.1/22 flags FIN ACK  on interface my-interface",
  "message_id": "106015",
  "log_level": "6",
  "product": "ASA",
  "event.tags": [
    "cisco",
    "asa",
    "example",
    "alternative"
  ]
}

[strikaco]

Just wanted to say this was immensely useful to me in constructing some complex user-defined types and alternative parsers I could not figure out how to define otherwise. The documentation for both is rather uncreative with its examples, so thank you for publishing this!