search

influxdata / go-syslog

Blazing fast syslog parser
MIT License
476 stars 69 forks source link

RFC 3164 #15

Closed marcosrmendezthd closed 4 years ago

marcosrmendezthd commented 6 years ago

Hi,

Any idea when that feature branch could be merged?

Thanks!

leodido commented 6 years ago

We are working on that. Not in our top priority right now.

Thanks for pointing this out.

marcosrmendezthd commented 6 years ago

Let me know if I can help. I'll be doing a fork in the mean time to get going. 😄

goller commented 6 years ago

@marcosrmendezthd that'd be great! @leodido started a branch here: https://github.com/influxdata/go-syslog/tree/feature/rfc3164

@leodido what needs to happen on that branch?

leodido commented 6 years ago

Tomorrow I will reserve some time to take stock of the situation (and probably push some code I have locally).

marcosrmendezthd commented 6 years ago

thank you @leodido !

leodido commented 6 years ago
marcosrmendezthd commented 6 years ago

@leodido the one thing i did want to comment on both the RFCs is to allow the timestamp parsing format to be configurable to support multiple layouts. that would be really awesome! some of the devices we're working on don't follow the strict standards. and this would solve the issues i'm seeing. this could be done as a separate PR.

thank you again for a great library!

leodido commented 6 years ago

@marcosrmendezthd i understand the issue with timestamp, particularly for the BSD-syslog format since its RFC (RFC3164) is not very strict and/or formal.

About the RFC5424 we could think about it but the RFC clearly mandates a RFC3339 micro format for them (cc @goller).

Another solution in this case could be to proxy almost standard messages through a relay (eg., rsyslog, syslog-ng) that converts them in standard ones.

marcosrmendezthd commented 6 years ago

understood. we are building the log receiver so forwarding won't be an option.

i've gone thru some of our packet captures and can only confirm RFC3164 deviations at this time; e.g. <166>Jul 06 2018 20:33:28: %ABC-1-234567: Some message here. i'll be going thru some more later on to confirm. :smile:

leodido commented 6 years ago

Thanks! Would be very helpful to collect some "common" BSD-syslog messages.

On Fri, Jul 13, 2018, 8:47 PM Marcos Mendez notifications@github.com wrote:

understood. we are building the log receiver so forwarding won't be an option.

i've gone thru some of our packet captures and I can only confirm RFC3164 deviations at this time; e.g. <166>Jul 06 2018 20:33:28: %ABC-1-234567: Some message here. i'll be going thru some more later on to confirm. 😄

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/influxdata/go-syslog/issues/15#issuecomment-404920484, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHU8_zrmW9yxmvaYoofuFe3FKY8pkiqks5uGOs-gaJpZM4VEvLB .

jtarchie commented 6 years ago

This is a library that supports date parsing natively from a string, rather than having to define formats. Would this help?

marcosrmendezthd commented 5 years ago

hi @leodido any updates on this branch?

russorat commented 5 years ago

FYI @robcowart

marcosrmendezthd commented 5 years ago

hi. any more progress on this?

steenstra commented 5 years ago

I would like to know if or when RFC 3164 will be supported.

Zapero commented 5 years ago

Any updates to this?

bastjan commented 4 years ago

Would be very helpful to collect some "common" BSD-syslog messages.

Some sample messages I've seen in the wild:

Citrix Netscaler:

<134> 11/28/2019:15:31:21 GMT netscaler1 0-PPE-0 : default TCP CONN_TERMINATE 17000000 0 :  Source 127.0.0.1:80 - Destination 127.0.0.1:25963 - Start Time 11/28/2019:15:30:06 GMT - End Time 11/28/2019:15:31:21 GMT - Total_bytes_send 1 - Total_bytes_recv 1

Cisco IOS:

00:00:46: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Feb 14 09:40:09.325: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Feb 14 09:40:09: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Mar  1 18:48:50.483 UTC: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) 
^                       ^- note the colon after the timestamp
\- Cisco timestamp extensions, the first '*' indicates that the clock is
   * unsynced, '.' if it is known to be synced
   https://github.com/syslog-ng/syslog-ng/blob/3a1bda0d9a9e42b5cd7e5a02ca05f5f896ef82b6/modules/syslogformat/syslog-format.c#L219

000019: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)
^- sequence number

Official Go syslog/ Unknown others:

<28>2019-12-02T16:49:23+01:00 host app[23410]: Test
    ^- rfc3339 timestamp

Unknown:

<13> Dec  2 16:31:03 host app: Test
    ^- note the space
russorat commented 4 years ago

Thanks for the additional info. This is something we still have on our backlog but we would be open to reviewing a PR if you'd like to contribute.

let4be commented 4 years ago

Any update? Telegraf cannot accept any logs from golang in syslog format, I've tried every library on the net - hopeless...

goller commented 4 years ago

@let4be yup, work is happening for RFC 3164 in pull request #27

As an aside, do you have some specific programs you want working?

let4be commented 4 years ago

I just want to be able to send logs from my golang code to telegraf, either via log/syslog or any other syslog logging available... Right now go-syslog seems to have trouble parsing logs sent by log/syslog

Currently nothing seems to work out of the box, so I opted out to directly sending logs to influxdb(their final destination)

goller commented 4 years ago

@let4be great ok. That format is:

<PRI>TIMESTAMP HOSTNAME TAG[PID]: MSG

Adding a log/syslog roundtrip test could be really nice.

Thanks for the info!

goller commented 4 years ago

@let4be We just released a v3 of this library that now includes RFC 3164 support. The parser is fairly strict at the moment, but, let us know your experiences. We have a system in place to extend the parser in order to relax its grammar if needed.