search

segmentio / logspout-fluentd

Logspout module for logging to fluentd.
8 stars 29 forks source link

logspout-fluentd fails to reconnect if it loses connectivity with fluentd #1

Open michaeltravisuk opened 6 years ago

michaeltravisuk commented 6 years ago

We're seeing issues with logspout-fluentd where connectivity is never re-established with fluentd if it is lost, what we see instead is continuous logging of broken pipe messages in stderr, which causes a large spike in CPU activity on that docker host:

2018/01/09 15:07:31 fluentd-adapter: write tcp 127.0.0.1:38830->127.0.0.1:24224: write: broken pipe

We're running logspout in a container on each of our Rancher hosts talking to a separate container running fluentd. This issue requires us to restart the logspout container to re-establish connection. I saw a similar issue raised for the syslog logspout adapter: https://github.com/gliderlabs/logspout/issues/107 in which they raised https://github.com/gliderlabs/logspout/pull/291 to resolve the issue by adding a configurable number of reconnection attempts.

We much prefer the richness of the logging using logspout-fluentd over syslog, so would it be possible to implement a similar fix to the fluentd adapter?

michaeltravisuk commented 6 years ago

Raised https://github.com/segmentio/logspout-fluentd/pull/2 to resolve the issue using code from https://github.com/gliderlabs/logspout/blob/master/adapters/syslog/syslog.go.

pilgrim2go commented 6 years ago

Any update on this ticket?