search

eht16 / python-logstash-async

Python logging handler for sending log events asynchronously to Logstash.
MIT License
186 stars 51 forks source link

Memory leak eridication #73

Closed PMazarovich closed 2 years ago

PMazarovich commented 2 years ago

We faced with steady increasing of RAM consumption in AWS ECS and this began to happen after adding python-logstash-async library. There is a known issue about Queue memory leak: https://bugs.python.org/issue43911 The simplest way to get rid of it - change Queue to PriorityQueue

eht16 commented 2 years ago

Nice, thank you!

Did you test that the memory consumption is lower by using PriorityQueue or do you only assume it?

I'm still testing it with and without to compare the memory consumption of both variants.

PMazarovich commented 2 years ago

@eht16 , yes, here is 2 graphics, 1 one is Queue, second one - PriorityQueue Tested with this simple code

def pr():
    for x in range(0, 1000000):
        queue.put("message, message, message, message, message, message, message, message, message")
    while not queue.empty():
        queue.get()
    print(queue.qsize())
pr()

1 2