search

fluent / fluent-logger-python

A structured logger for Fluentd (Python)
http://fluentd.org/
Other
444 stars 138 forks source link

using fluent-logger-python with eventlet #183

Open jshen28 opened 2 years ago

jshen28 commented 2 years ago

Hello,

I saw a surprising memory usage growth (see this issue). But when I remove monkey_patch looks like memory usage back to normal. Does fluent-logger-python compatible with eventlet?

arcivanov commented 2 years ago

I don't know about eventlet, but I have been using it with gevent in production for about 4 years now with no problems.

jshen28 commented 2 years ago

I saw fluent-logger uses threading.Local to store last errors, but does it make sense when it is running in an eventlet? I found, although it could be wrong observation, by removing threading.local, memory usage could be surprisingly reduced.

arcivanov commented 2 years ago

The question is how does eventlet monkey-patch the threading.local if at all? You'll also see that we're using threading.Lock. If eventlet is leaking memory through threading.local because it didn't monkey it correctly or if you're leaking eventlets that are not being garbage collected and thus retain local state, then there is not much Fluent logger can do about it, sorry.

jshen28 commented 2 years ago

I understand, but I am still interested why last_error is stored and how can I take advantage of it. Because I am afraid this may not be easy to solve for eventlet, so if last_error is not used, I am thinking to fork the repo and maintaining a custom version...

arcivanov commented 2 years ago

The reason for a thread local last error is because the same sender is global, can be shared by multiple threads as it's stateless, except for last error. Furthermore, the same behavior in AsyncSender allows the last error to be attributed to the invoking thread. It's not the best design but it's been there for a long time. Judging by eventlet/eventlet#741, it has been confirmed to be an eventlet bug.

I am thinking to fork the repo and maintaining a custom version...

https://www.youtube.com/watch?v=BvK6KsLkPUs