davidelang
commented
9 years ago when running into issues like this, run it with -v to get details
In this case, I see:
liblognorm: 46:trying parser for field 'sensor_host': 0x7fda6332e870 liblognorm: parser returns 0, parsed 10 liblognorm: potential hit, trying subtree liblognorm: 56: prefix compare ' ', ' ' liblognorm: 57: prefix compare succeeded, still valid liblognorm: 57:trying parser for field 'tag': 0x7fda6332ea00 liblognorm: parser returns 0, parsed 9 liblognorm: potential hit, trying subtree liblognorm: 66: prefix compare ':', ':' liblognorm: 67: prefix compare ' ', ' ' liblognorm: 68: prefix compare succeeded, still valid liblognorm: 68:trying parser for field 'cef': 0x7fda63331870 liblognorm: parser returns 0, parsed 18446744073709551554 liblognorm: potential hit, trying subtree liblognorm: 6: prefix compare succeeded, still valid liblognorm: 6 no field, trying subtree char ' ': (nil) liblognorm: 6 returns 163
it looks like the parser is running off the end of the string ("parsed 18446744073709551554") and then for some reason it's finding a space and comparing it to the end of the pattern
adding %rest:rest% to cef.db returns the following
echo "local0 info: 2015-05-05T03:31:04.364405+00:00 sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3 " | /usr/lib/lognorm/lognormalizer -e json -r cef.db |jq . { "rest": " info: 2015-05-05T03:31:04.364405+00:00 sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3 ", "cef": { "DeviceVendor": "Vendor", "DeviceProduct": "Product", "DeviceVersion": "Version", "SignatureID": "Signature ID", "Name": "some name", "Severity": "Severity", "Extensions": { "a": "field1", "bb": "this is a value", "cc": "field 3 " } }, "tag": "syslogtag", "sensor_host": "sysloghost", "date": "2015-05-05T03:31:04.364405+00:00", "priority": "info", "facility": "local0" }
so it looks like the cef parser is moving the wrong direction through the string at some point. from the lengths of things, it looks like it's moving a pointer in the wrong direction as it's parsing everything before the "Extensions" section (62 characters from CEF through | and the space before info is 63 characters back)
vdesabou
ontholerian
janmejay
First and foremost thank you for taking the time to develop this feature. This will make parsing CEF a whole lot more efficient for us. I love where this project is going as a whole and appreciate the responsiveness.
I'm still experimenting with the new CEF parser but I wanted to report a couple issues I've seen so far.
1) This one is relatively straight forward. The first character after the last pipe in the mandatory fields is ignored. In your example you have a space there, so it was immediately obvious why it was happening on our logs.
$ echo "CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3" | lognormalizer -e json -r cef.db | jq .cef.Extensions { "a": "field1", "bb": "this is a value", "cc": "field 3" }
2) This one is a little strange to me. We usually put our syslog template as a prefix for all of our rules. It took me a while to narrow down why the CEF parser wasn't working here, but I narrowed it down to the date field. Example below:
This fails with our standard prefix: $ cat cef.db prefix=%facility:word% %priority:char-to:\x3A%: %date:date-rfc5424% %sensor_host:word% %tag:char-to:\x3A%:\x20 rule=:%cef:cef%
$ echo "local0 info: 2015-05-05T03:31:04.364405+00:00 sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3" | lognormalizer -e json -r cef.db | jq . { "originalmsg": "local0 info: 2015-05-05T03:31:04.364405+00:00 sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3", "unparsed-data": "CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3" }
It works fine when I use rest instead of CEF for the type: $ echo "local0 info: 2015-05-05T03:31:04.364405+00:00 sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3" | lognormalizer -e json -r cef.db | jq . { "cef": "CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3", "tag": "syslogtag", "sensor_host": "sysloghost", "date": "2015-05-05T03:31:04.364405+00:00", "priority": "info", "facility": "local0" }
Removing the date from the parser and the input resolves the issue. Note that I tried changing date to another type like "word" and it had no effect.
$ cat cef.db prefix=%facility:word% %priority:char-to:\x3A%: %sysloghost:word% %syslogtag:char-to:\x3A%:\x20 rule=:%cef:cef%
$ echo "local0 info: sysloghost syslogtag: CEF:0|Vendor|Product|Version|Signature ID|some name|Severity|aa=field1 bb=this is a value cc=field 3" | lognormalizer -e json -r cef.db | jq . { "cef": { "DeviceVendor": "Vendor", "DeviceProduct": "Product", "DeviceVersion": "Version", "SignatureID": "Signature ID", "Name": "some name", "Severity": "Severity", "Extensions": { "a": "field1", "bb": "this is a value", "cc": "field 3" } }, "syslogtag": "syslogtag", "sysloghost": "sysloghost", "priority": "info", "facility": "local0" }