RFC5424 structured data parsing

[cybermedi]

Base on Jochen recommendation https://groups.google.com/forum/#!searchin/graylog2/5424/graylog2/KO91vcZIOXo/vcEPEwAT6e4J

I would like to create issue:

I have an application which creates log in RFC5424 and send them to my central rsyslog server. Logs are resend to graylog2 via syslog protocol. Graylog2 runs local INPUT tcp syslog on 10514. Configuration of rsyslog resending looks like:

$template GRAYLOGRFC5424,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %STRUCTURED-DATA% %msg%\n"
*.* @@172.100.100.100:10514;GRAYLOGRFC5424

all logs are received by graylog2 without any issue but they are not parsed properly. They are parsed like basic RFC5424 message. Structured data are handled like "message".

the full message looks like:

<190>1 2015-01-06T20:56:33.287Z app-1 app - [mdc@18060 ip="::ffff:132.123.15.30" logger="{c.corp.Handler}" session="4ot7"  user="cybermedi@yahoo.com" user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/7.1.2 Safari/537.85.11"] User page 13 requested

I recieve message fields like: application_name app

facility local7

level Info [6]

message [mdc@18060 ip="::ffff:132.123.15.30" logger="{c.corp.Handler}" session="4ot7" user="cybermedi@yahoo.com" user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/7.1.2 Safari/537.85.11"] User page 13 requested

source app-1

but there should be other ones based on structured data. Like: ip, logger, session, user, user-agent and in the message should be just: User page 13 requested

[lennartkoopmann]

Which Graylog2 version are you running? @bernd fixed something regarding this recently.

[cybermedi]

Hi Lennart, I have 0.92.3 (102a59f) (Moose) Regards, Martin

1. 1. 2015 v 2:47, Lennart Koopmann notifications@github.com:

Which Graylog2 version are you running? @bernd fixed something regarding this recently.

— Reply to this email directly or view it on GitHub.

[dfch]

I can confirm this behaviour (especially when you pass a NILVALUE (-), this is also treated as part of the message) (also running 0.92.3). In addition, the field MSGID seems to be treated as part of the message as well.

Message on the wire:

<128>1 2015-01-11T16:35:21.335797+01:00 s000000.dfch.biz - - - - tralala

... will have Graylog2 recognise the message field as:

- - tralala

See 6.5. Examples for an example.

Note: the VERSION field is not recognised or extracted at all.

I used modified syslog sender for testing.

[bernd]

@cybermedi There are several issues unfortunately.

1. Your rsyslog template is wrong. The %MSGID% is missing. See our docs for a template example. (our example was also wrong until yesterday...)
2. There are two spaces after the "session" key in your example. Is that a copy & paste error? If not, rsyslog might generate a structured data block that is not valid according to the RFC.
3. The syslog parser we are using does not support whitespace in structured data values like the "user-agent" one.

I am currently investigating if we can fix the whitespace-in-value parsing issue in our syslog parser.

[bernd]

@dfch This also seems to be an issue with the syslog parser. As said in the previous comment, I will investigate.

[cybermedi]

OK I added msgid to template. And it seems that that there is still something wrong with parsing. Structured data was not parsed, and the message begins with "-" (i think it is the msgid). In my message I have app-name I do not have procid and msgid, that's why raw messages contains: "app - - [mdc@18060...". The message beings: "- [mdc@18060..." what is interesting that %STRUCTURED-DATA% when they are empty are sometimes replaced by space and sometimes not. But finally the message contains two or three spaces between msgid and msg ex: sshd 4988 - Accepted public key for sudo - - cybermedi : TTY=pts/0

OH I do not how to show it here, the first example has two spaces and the second three. Maybe I will need to have two templates for messages with structured data and one without. Maybe this issue should be fixed in rsyslog.

[bernd]

@cybermedi The next beta (beta.2) will have a fix for the structured data parsing.

It will also remove the structured data from the message field.

@dfch If there is a NILVALUE for the structured data like you posted, the "- -" will still be in the message field. This is a shortcoming in the syslog parser which we cannot fix for 1.0 unfortunately.

See Graylog2/graylog2-server#872 for the pull request.

I am closing this ticket now. Please re-open if you still have issues with the next beta. Thank you for the report!

[cybermedi]

Hi Bernd, I deployed 1.0.0-9.rc.2. 1) you mentioned the issue with parsing white space. Is it fixed? 2) The structured data is correctly removed from "message" 3) The structured data is still not parsed. I use the corrected template configuration:

$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%n"
*.* @@10.19.10.1:5514;GRAYLOGRFC5424 # Graylog forwarding

[bernd]

Hey @cybermedi, thanks for the feedback!

Did you enable the "Expand structured data?" option on the input?

[cybermedi]

Hi, yes

Martin

1. 2. 2015 v 16:09, Bernd Ahlers notifications@github.com:

Hey @cybermedi, thanks for the feedback!

Did you enable the "Expand structured data?" option on the input?

— Reply to this email directly or view it on GitHub.

[bernd]

There is an issue if the "Expand structured data?" option is enabled. Without this, the structured data is parsed.

Thanks for the report!

[bernd]

I created a pull request in #965.

@cybermedi As a workaround you can disable the "Expand structured data" option.

[bernd]

@cybermedi The fix (#965) got merged and will be availablein an upcoming snapshot or RC.

[cybermedi]

thanx it works now.

[bernd]

Great, thanks for the feedback!