search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.68k stars 3.42k forks source link

Support syslog-ng #153

Closed tomwilkie closed 4 years ago

tomwilkie commented 5 years ago

https://twitter.com/PCzanik/status/1074996519638568961

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.19/administration-guide/36#TOPIC-1094599

https://www.syslog-ng.com/community/b/blog/posts/bulk-mode-message-sending-to-elasticsearch-with-syslog-ng-http-destination

madmaze commented 5 years ago

Would you envision this capability to be built into loki itself or an external translator/forwarder?

tomwilkie commented 5 years ago

Haven't given it much thought TBH! We're looking to add fluend support, so perhaps that can act as the translator?

madmaze commented 5 years ago

I'll prototype a translator proof-of-concept in the coming days, but I think ultimately it would be nice to have ingest modules/plugins.

cypherfox commented 5 years ago

Any update on this? Can I see the translator code? Never mind if it is rough. I just want to get the other ducks in line by deciding where to use syslog(-ng) and where to look for alternatives when logging non-k8s sources.

daixiang0 commented 5 years ago

@tomwilkie hi, any process about this?

odormond commented 5 years ago

It's currenlty impossible to ingest lines from syslog properly even with pipeline stages without reconfiguring syslog itself. The problem is that the default timestamp at the beginning of the lines looks like Jun 20 14:54:51 which even with a proper timestamp stage format will be parsed as time=0000-06-20T 14:54:51Z which will fail to encode: msg="error encoding batch" error="timestamp: &types.Timestamp{Seconds: -62152391109,\nNanos: 0,\n} before 0001-01-01"

Note: I opened an explicit ticket for the date parsing issue: https://github.com/grafana/loki/issues/692

pbadenski commented 5 years ago

Is this possible now considering https://github.com/grafana/loki/pull/760 has been merged?

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

eagle1981 commented 5 years ago

+1

cyriltovena commented 5 years ago

With our new fluentbit output plugin, it should be possible to capture syslog via this official input plugin https://fluentbit.io/documentation/0.12/input/syslog.html