Support for Key-Value Parser

[crackytsi]

Hello, I have logs (from trendmicro console) in the following format (beside from the prefix) Key<space...="" Key<space...=""

Unfortunately the key-value parser does not support spaces in they key and value markings via quotations. Do I miss something, or is it possible to extend the key-value parser for this format?

Here a raw event: <66>Mar 1 02:43:31 Hostname TMCM: SLF_INCIDENT_EVT_VIRUS_FOUND_QUARANTINE_SUCCESS Security product="ScanMail for Microsoft Exchange" Security product node="HE105647" Security product IP="1.2.3.4" Event time="06.03.2018 01:36:41 (UTC)" Virus="TSPY_HPLOKI.SM1" Infected file="PLS QUOTE PO # BD007362.zip" File path="SMTP" Action taken="Quarantine" Result="Quarantine successfully" Infection destination="info@user.de;" Infection destination IP="1.2.3.4" Infection source="prvs=596e3ff46=UserA@xyz.com.sg;" Infection source IP="" Destination IP="" Source IP="" Domain="internal.dom" ScanMethod="Real-time Scan" User="N/A" Managing server entity="Server" Event time (local)="01.03.2014 02:36:41"

[elcamlost]

I found a workaround for it, which helped me in my case (nginx error log)

Use repeat parser to populate key-value json attribute like these

version=2

rule=: %-:string-to{"extradata":"product"}% %
    {"name":"kv", "type":"repeat",
    "parser":[
               {"type":"char-to", "name":"key", "extradata":"="},
               {"type":"literal", "text":"=\""},
               {"type":"char-to",  "name":"value", "extradata":"\""}
             ],
    "while":[
               {"type":"literal", "text":"\" "}
            ]
    }%

It gives you json

{ "kv": [ { "value": "ScanMail for Microsoft Exchange", "key": "product" }, { "value": "HE105647", "key": "Security product node" }, ... ] }

And then in rsyslog.conf you can

...
    action(type="mmnormalize" rulebase="/path/to/rulebase")
    foreach ($.i in $!kv) do {
        if ($.i!key == 'product') then {
            set $!_product = $.i!value;
        } else if ($.i!key == 'Infection source') then {
            set $!_inf_source = $.i!value;
        } ....
    }
    unset $!kv;
    call   ruleset; 

It's not good and we still need key-value parser in liblognorm, but better than nothing.

[soulbreak]

Hi,

I found another solution which is working for a mixture of quoted value and unquoted value. Not sure this option quoting.mode was already released when the subject was opened.

~ cat key_value.rb
version=2

rule=:%
   {"name":"kv", "type":"repeat",
    "option.permitMismatchInParser":true,
    "parser":[
               {"type":"char-sep", "name":"key", "extradata":"="},
               {"type":"literal", "text":"="},
               {"type":"string", "quoting.mode":"auto","name":"value" }
             ],
    "while":{
                             "type":"alternative", "parser": [
                                {"type":"literal", "text":" "},
                         ]
             }
    }%

~ echo 'type=unquoted type="very quoted"' |  /usr/bin/lognormalizer -r key_value.rb
{ "kv": [ { "value": "unquoted", "key": "type" }, { "value": "very quoted", "key": "type" } ] }