Support multiline in syslog input

[jsoriano]

multiline is currently not supported in syslog input, there was a question about that in discuss.

We probably need to keep in mind for this input that lines can include the hostname and the username as preffix (for example if the log is sent using the logger command).

[ph]

@jsoriano Yes, we need to add support for that. @kvch is doing a lot of refactoring around our readers in Filebeat to make them reusable and allowing us to support this use case.

[daveman1010221]

Hey, all. Just a thought, but maybe you could log a message that says you're ignoring all multiline options in the config for type: syslog? Would have saved me a day and a half of debugging before I found this...

[kvch]

I am sorry you had to debug this for a day. The list of currently supported config options can be found here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-syslog.html

Unfortunately, our configuration validation is not able to detect such problems yet. We are adding a configuration schema to avoid such issues soon.

[GeorgeGkinis]

I am trying to work around this issue by using a dissect filter to remove leading fields like timestamp, host e.t.c.

I use the log input type as all logs are available via SMB.

Where i am stuck is at applying multiline after dissect has taken place.

Maybe create a distinct multiline processor? It could solve my issue by allowing me to control the order of execution. This way there can be more fine grained control and many possibilities to solve this type of issues.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[krka01]

Is there any active work ongoing regarding this issue? I have also run into this issue and spend several hours debugging the problem before I realized it was due to multiline syslog messages. It seems like the message are recieved as it should in the message field, but once I try to use the copy_fields, rename or dissect processors the problem starts.

[Lavaburn]

Any plans to implement this or are there any workarounds to process multiline over syslog?

[ash-darin]

If I may make a suggestion for this topic:

Issues:

a): Vendors broadly disregard the syslog standards in general, newlines are common occurrance. b): Filebeat spams my log with the information that message X "can not be parsed according to RFC"

Suggestion:

A "dummy" mode for syslog over TCP, that throws no error message and simply assumes this (multiline like):

Everything between "'^<\d{1,3}>'" is one syslog message. No further parsing except syslog priority. The rest is to be parsed elsewhere.

I think that would help quite a bit. (I put up a seperate rsyslog and did this with filestream/multiline parser).

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!