search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.06k forks source link

Graylog 3.2: rsyslog parsing errors #8780

Open fwetzler98 opened 4 years ago

fwetzler98 commented 4 years ago

When parsing rsyslog messages the built-in field extraction (not with extractors or pipeline) provides wrong content

Expected Behavior

When getting rsyslog messages the fields (timestamp, level, facility, message ...) should be extracted correctly (or NO extraction)

Current Behavior

Graylog told me of Indexing errors, so I investigated. Graylog (elasticsearch) expected a Number in field "level" but was filled with a string ("error", "warning", ...) which is correct! Additionally the field "message" (in same message) was not parsed correctly. Parsing stopped after first backslash ("\") which can be seen in extracted field "msg".

Possible Solution

  1. Parsing field "message" should contain whole message, not stopping at first backslash
  2. Field "level" should renamed to "severity" (analog to rsyslog naming convention)

Steps to Reproduce (for bugs)

In our case it was found per accident. Telegraf was not configured correctly, so it (and docker daemon) threw these messages. No extractor and no pipeline used. Only a static field produced by input.

1. Content /var/log/messages: Aug 12 09:33:00 infra-dev-test-fixed-2 dockerd: time="2020-08-12T09:33:00.370576643+02:00" level=error msg="Handler for GET /v1.21/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again." Aug 12 09:33:00 infra-dev-test-fixed-2 telegraf: 2020-08-12T07:33:00Z E! [inputs.docker] Error in plugin: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

2. Field contents in Graylog (for both messages see above) first message: application_name="dockerd" facility="system daemon" level=6 message=(see messages above) msg=Handler for GET /v1.21/services returned error: This node is not a swarm manager. Use \ time=(ok) timestamp=(ok)

second message: application_name="telegraf" facility="system daemon" level=6 message=2020-08-12T12:46:00Z E! [inputs.docker] Error in plugin: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. (no field msg) timestamp=(ok) (no field "time")

Context

Your Environment

Single Instance in docker Vsphere-VM (6.7), 16 GB, 4 CPU

fwetzler98 commented 4 years ago

Sorry, I am confused ... The field "level" in Graylog-syslog corresponds to "severity" in rsyslog ... perhaps rename field ...?

mikkolehtisalo commented 3 years ago

Level comes from GELF , and should be a number. Syslog also wants numbers for severity, which sometimes is shown to users as textual values (for example in /var/log/messages). Graylog maps severity to level automatically. As far as I can see Graylog parses this correctly.

In syslog the backslash should be escaped, and I'm pretty sure Graylog would parse that correctly. There is probably something else going on here. It is known that rsyslog and jsonr at least have encoding issues.