Ensuring logs make it to Graylog server

mtotheikle commented 9 years ago

I'm just getting started with Graylog and fluent but is there any way to ensure logs get delivered to the Graylog server even if the Graylog server is temporarily unreachable? Thanks!

emsearcy commented 9 years ago

fluentd should take care of this automatically. out_gelf is a "buffered plugin", which means that it maintains an internal queue of "chunks" (a chunk is a collection of events), which it buffers (by default) to memory, and it tries to send these to the Graylog server but will retry if it encounters an error. (See http://docs.fluentd.org/articles/output-plugin-overview#list-of-buffered-output-plugins for more info on tuning buffering including the size of the buffer and how long to retry.)

However, it would be great if you could confirm this behavior is working for you, and let me know if you seem to be loosing any messages!

mtotheikle commented 9 years ago

Thank you @emsearcy for confirming what I was expecting.

I can't seem to get the expected behavior out of this though. I have tested by shutting down an input within the Graylog cluster. This input is what my Fluent configuration is set to write to and when this input is not available it seems like the generated logs during that period are lost since when the input comes back up no information during that period is available within Graylog.

My configuration file looks like this:

<source>
    type tail
    format nginx
    path /var/log/nginx/access.log
    tag graylog2.nginx
  </source>
  <match graylog2.**>
    type copy
    <store>
      type gelf
      host x.x.x.x
      port 12201
      flush_interval 5s
      buffer_type file
      buffer_path /var/log/td-agent/buffer/nginx
    </store>
    <store>
      type stdout
    </store>
  </match>

I've also attempted a in memory buffer like so:

<source>
    type tail
    format nginx
    path /var/log/nginx/access.log
    tag graylog2.nginx
  </source>
  <match graylog2.**>
    type copy
    <store>
      type gelf
      host x.x.x.x
      port 12201
      flush_interval 5s
    </store>
    <store>
      type stdout
    </store>
  </match>

emsearcy commented 7 years ago

Can you try using TCP to connect to Graylog? (protocol tcp)

UDP is stateless (fire and forget) so there isn't any error checking to see if a GELF packet was received or not.

KevDBG commented 6 years ago

Hello,

It's works with TCP, but there is a new problem comment here https://github.com/emsearcy/fluent-plugin-gelf/issues/28

emsearcy / fluent-plugin-gelf

Ensuring logs make it to Graylog server #7