search

deis / fluentd

Log shipping mechanism for Deis Workflow
https://deis.com
MIT License
10 stars 23 forks source link

Document how to drain to an external log archive #27

Closed krancour closed 8 years ago

krancour commented 8 years ago

Draining logs to an external archive for long term storage, indexing, analysis, etc. was a recurring concern for v1.x users operating "stateless clusters." It became even more prevalent when we implemented the in-memory ring buffer as the default log storage in logger-- which further encouraged users to ship their logs someplace else.

The Drain interface has long since been removed from the logger, and I believe @jchauncey has stated that the right thing to do here is to configure fluentd to forward logs directly to such an archive (e.g. Papertrail or Splunk) in addition to the logger component... but have we documented how that is done? I cannot find anything to this effect here in the v2 docs.

jchauncey commented 8 years ago

Right now the we really can only drain to syslog endpoints and that is documented here - https://docs-v2.readthedocs.io/en/latest/managing-workflow/platform-logging/ and here https://github.com/deis/fluentd#fluent-plugin-remote_syslog but it could probably be made clearer.

jchauncey commented 8 years ago

Well i take that back we can also drain to elastic search so i should document that too

dmcnaught commented 8 years ago

Are there any plans to support drain to splunk in v2? Thanks

jchauncey commented 8 years ago

@dmcnaught not at this time. IIRC splunk can accept data via syslog so you could configure fluentd to forward all the data to a splunk forwarder via the syslog mechanism.

jchauncey commented 8 years ago

closing this since I have updated the docs to be more inline with our currently offered plugins