fluent / fluent-logger-python

A structured logger for Fluentd (Python)
http://fluentd.org/
Other
444 stars 138 forks source link

add discard_logs_on_reconnect_error in asyncsender #176

Open enjoy-binbin opened 3 years ago

enjoy-binbin commented 3 years ago

issue: https://github.com/fluent/fluent-logger-python/issues/175

When using asyncsender, there is a queue holds the logs to be send:

This is my solution, it doesn’t look particularly good. so feel free to close this if un-needed

coveralls commented 3 years ago

Coverage Status

Coverage remained the same at 100.0% when pulling 6995dbb606e22a16f926e882ef0a5d573b250f43 on enjoy-binbin:connect-hang into ace80f4c2bd0020fc16891440243663c612dd01e on fluent:master.

coveralls commented 3 years ago

Coverage Status

Coverage decreased (-1.07%) to 98.928% when pulling bb8efb8930c5ed8ece8dbad69ae978f7a51c642d on enjoy-binbin:connect-hang into ace80f4c2bd0020fc16891440243663c612dd01e on fluent:master.

arcivanov commented 3 years ago

I'll review it at depth when I have time, but I'm inclined not to accept this change.

Unless I'm missing something the error should lead to a reconnect and resending of the logs. Clearing the queue will cause the loss of perfectly good logs due to a simple reconnect issue, which would not be acceptable.

enjoy-binbin commented 3 years ago

Yep Not problem. I understand it is risky

Or maybe we should just print some logs to help the user for debug

arcivanov commented 3 years ago

We can consider a tunable setting which would "discard logs on reconnect" and defaults to False. Reconnects are a part of life and discarding logs isn't useful in most cases IMO.

I tend to make decisions against arbitrary loss of data.

enjoy-binbin commented 3 years ago

Right. I can give a try. Thanks @arcivanov

enjoy-binbin commented 3 years ago

@arcivanov I add a discard_logs_on_reconnect_error setting default false. Maybe not good enough