search

tagomoris / fluent-plugin-secure-forward

Other
140 stars 30 forks source link

EPIPE Exception causes detached connection state which never get re-connected by node watcher. #14

Closed BitFlippedCelt closed 9 years ago

BitFlippedCelt commented 9 years ago

Warn level errors show in the log when this error condition initially occurs. The node watcher has no logic relating to the detatched? state and therefore never attempts to reconnect the session.

2015-02-11 08:31:33 -0600 [warn]: temporarily failed to flush the buffer. next_retry=2015-02-11 08:31:34 -0600 error_class="Errno::EPIPE" error="Broken pipe" plugin_id="object:3faf4f80bb80" 2015-02-11 08:31:33 -0600 [warn]: Failed to send messages to ingest.data.goldenfrog.com, parging. error_class=Errno::EPIPE error=#

These errors are then followed by a series of failures sending and ultimately it appears that fluentd stops trying to send and we loose events as they are pruned out based on retry rules.

I have submitted a pull request with changes that address this issue: https://github.com/tagomoris/fluent-plugin-secure-forward/pull/13

tagomoris commented 9 years ago

I've just merged #13. Is it fix this issue completely? Or any other actions are required?

BitFlippedCelt commented 9 years ago

As far as I can tell the issue has been resolved by #13, we have been running for a little over 2 weeks with no further issue across 20 or so updated secure-forwarders.

tagomoris commented 9 years ago

Thanks to confirm my comment!