search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.44k stars 1.07k forks source link

VMware-logs incorrectly detected as RFC5424 #1739

Open jeff42 opened 8 years ago

jeff42 commented 8 years ago

Hi, the logs from vmware are incorrectly detected as RFC5425. vmvware is sending messages like this:

<166>**2**016-01-29T15:20:31.960Z srv01esx70 Vpxa: [FF8DFB70 verbose 'VpxaHalCnxHostagent' opID=WFU-ba76b75e] [WaitForUpdatesDone] Completed callback due to the digit immediatly after the it is incorrectly detected as RFC5424. the regex at graylog2-server/graylog2-server/src/main/java/org/graylog2/inputs/codecs/SyslogCodec.java line 63 is `"<\\d+>\\d.*", Pattern.DOTALL` which matches every digit. According to RFC5424 the VERSION field is a one to three digit number never starting with a 'zero' which must be followed by a space. Improving this regex to this: `"^<\\d{1,3}>[1-9]\\d{0,2}\s.*"` should fix this issue. Please check the escaping since I'm not familiar with with Java Patterns. Best Regards. Jörg
joschi commented 8 years ago

@jeff42 Thanks for bringing this up! Could you please provide one or two more Syslog messages from the VMWare product to verify against?

jeff42 commented 8 years ago

@joschi VMware is just an example, I'll fetch some more. My suggesstion is more general. if you read RFC5424 (https://tools.ietf.org/html/rfc5424#section-6) you will find this

      HEADER          = PRI VERSION SP TIMESTAMP SP HOSTNAME
                        SP APP-NAME SP PROCID SP MSGID
      PRI             = "<" PRIVAL ">"
      PRIVAL          = 1*3DIGIT ; range 0 .. 191
      VERSION         = NONZERO-DIGIT 0*2DIGIT

which means a RFC5424-Syslog message must start with the Priority followed by the version and a space character. VERSION must be a number with one to three digits and must not start with zero. adjusting the static pattern to detect RFC5425 conform messages would automaticly stop applying wrong filters to vmware messages.

Examples: 

<166>2016-01-29T16:37:46.644Z srv01esx27.corp.local Hostd: --> }
<166>2016-01-29T16:33:30.567Z srv01esx24.corp.local Hostd: --> "992"
<166>2016-01-29T15:49:19.934Z srv01esx29.corp.local Vpxa: [6269EB70 verbose 'VpxaHalCnxHostagent' opID=WFU-bea1521f] [WaitForUpdatesDone] Received callback
<166>2016-01-29T15:49:19.934Z srv01esx29.corp.local Vpxa: [6269EB70 verbose 'VpxaHalCnxHostagent' opID=WFU-bea1521f] [WaitForUpdatesDone] Received callback
<166>2016-01-29T15:49:19.926Z srv01esx29.corp.local Vpxa: [6269EB70 verbose 'VpxaHalCnxHostagent' opID=WFU-c04d02e0] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 7313825 to 7313826 (at 7313825)

Best Regards, Jörg