search

philhagen / sof-elk

Configuration files for the SOF-ELK VM
GNU General Public License v3.0
1.5k stars 277 forks source link

add RFC5424 (IETF) syslog format handling #149

Open philhagen opened 5 years ago

philhagen commented 5 years ago

LS provides separate RFC5424 grok parsers[1], meaning conditional support is needed for that, in conjunction with necessary field normalization.

[1] https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/linux-syslog

philhagen commented 5 years ago

this may be more complicated than it at first looks... lots of grok hackery

https://github.com/logstash-plugins/logstash-input-syslog/issues/15

Melantrix commented 5 years ago

@philhagen I have received another message from Synology, which clarifies the <136> integer at the beginning of the IETF message. this is their response:

After further investigation by our developers, they found that IETF format is designated in RFC 5424.

There is also RFC 5425, RFC 5426 and RFC 6587 designation of IETF format which split into TLS, UDP and TCP protocol.

According to RFC 6587 Section 3.4.1, this number refers to log length. It's function is for splitting up multiple logs under TCP connections. And accordingly, Log Center's syslog-ng framework adheres to RFC standard.

What might be a possibility is that the log aggregator in question that you are using may not be adhering to RFC standard. Could you kindly double check that to be sure?

Thank you!

Best regards,

If i understand correctly it is part of a 'sub' RFC, so they have the RFC police on their side :P

i guess it's something to take into account when parsing the IETF messages.

philhagen commented 5 years ago

This is still something that would likely need to be raised with the Logstash developers. I'm not likely going to be able to maintain a processing pipeline that is forked from their codebase. SOF-ELK relies on the Logstash syslog processor. I'll see if there is anything I can do within our scope, but it does not look promising.

While they are correct that this technically does adhere to an RFC definition, it is not commonly used - I have never seen a log shipper that uses it. Similarly, HTTP+STARTTLS is defined in an RFC, but I'd be shocked if any browser or server software supports it.

Melantrix commented 5 years ago

Okay, thanks for replying. Just to be clear, you are saying to raise this also with the logstash devs? Or should i wait to see if you can check this within the SOF-ELK scope?

philhagen commented 5 years ago

I don't plan to raise the issue, as it's not something I'm directly experiencing. I'll see if there is anything that can be done locally under the context of this issue, but initial research is not promising.

SMAPPER commented 5 years ago

Are you sure this is not the syslog PRI? A number such as <132> at the beginning of syslog is usually a mathematically computed syslog PRI.

On Mon, Mar 25, 2019, 7:42 AM Phil Hagen notifications@github.com wrote:

I don't plan to raise the issue, as it's not something I'm directly experiencing. I'll see if there is anything that can be done locally under the context of this issue, but initial research is not promising.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/philhagen/sof-elk/issues/149#issuecomment-476179747, or mute the thread https://github.com/notifications/unsubscribe-auth/ALPAz3rlA5gEuNoNxgM2ukdI3O-iGKFvks5vaMRGgaJpZM4bRgiz .

philhagen commented 5 years ago

there are two fields at play - one is definitely the PRI and he totally handle that. The other is a mystery integer that may or may not be the message length. If you look at the screenshot on tagged ticket #144 (https://user-images.githubusercontent.com/4669661/52183368-90c19a80-2807-11e9-8976-0d4d04b334cf.png), you'll see a 136 prefix. The 1 prefix is a "version" field, provided by the IETF syslog variant. Even the Logstash standard grok syntaxes don't seem to address the supposed length.

SMAPPER commented 5 years ago

That's a nightmare. Even if it is RFC standard. :)

I don't think Logstash will address this as there are ways to handle it already albeit ugly ones. The truth is, if this is how Synology is logging, they need to update how they send logs.

Here's a possible solution with Logstash:

Use aggregate plugin to combine multiple logs back together Use ruby plugin to add tags identifying if the length has been met

Sincerely,

Justin Henderson (312) 857-5755 Systems and Security Architect GSE # 108, Cyber Guardian Red / Blue http://www.linkedin.com/in/justinhenderson2014/

On Mon, Mar 25, 2019 at 8:43 AM Phil Hagen notifications@github.com wrote:

there are two fields at play - one is definitely the PRI and he totally handle that. The other is a mystery integer that may or may not be the message length. If you look at the screenshot on tagged ticket #144 https://github.com/philhagen/sof-elk/issues/144 ( https://user-images.githubusercontent.com/4669661/52183368-90c19a80-2807-11e9-8976-0d4d04b334cf.png), you'll see a 136 prefix. The 1 prefix is a "version" field, provided by the IETF syslog variant. Even the Logstash standard grok syntaxes don't seem to address the supposed length.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/philhagen/sof-elk/issues/149#issuecomment-476202743, or mute the thread https://github.com/notifications/unsubscribe-auth/ALPAz4zZOEEJT9xef8RLbRlf6jMiwMinks5vaNKWgaJpZM4bRgiz .