mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.4k stars 531 forks source link

Graylog native go plugin #1924

Closed xrl closed 8 years ago

xrl commented 8 years ago

I needed to get logs out of the docker daemon and the most attractive option was graylog. Problem is that my version of docker (1.9) could not disable compression and the simple graylog decoder was not going to be sufficient. So I tackled the native plugin. This should be a useful addition for other heka/docker users.

rafrombrc commented 8 years ago

I know I said that I was open to including this in the Heka core, but I'm having second thoughts. I don't really want to add additional dependencies unless it's for an extremely common use case. I'd rather see this packaged up as a separate package that can easily be included in a Heka build using the plugin_loader.cmake stuff.

Also, for future reference, if a plugin is to be included in the Heka core, it needs to go in the changelog, and the documentation needs to be updated.

xrl commented 8 years ago

Thanks for the feedback! I think graylog is a popular format, it's included as one of the default log drivers in the docker daemon (which is why I wrote this) and it's a default plugin in logstash.

There's already a graylog decoder in heka core but it's not broadly useful because it can't do the standard decompression you'll see coming out of graylog connections.

If it's any consolation, the included dependency is a very simple one and does not itself include other dependencies. It makes heavy use of the go stdlib.