search

WojciechMula / pyahocorasick

Python module (C extension and plain python) implementing Aho-Corasick algorithm
BSD 3-Clause "New" or "Revised" License
914 stars 122 forks source link

memory leak issue exists in 1.4.4 #183

Open pombredanne opened 1 year ago

pombredanne commented 1 year ago

Originally posted by @Azzonith in https://github.com/WojciechMula/pyahocorasick/issues/81#issuecomment-1157813973_

        Hello,

Can confirm memory leak issue exists in 1.4.4 and it might have something to do with pickling, tracemalloc output:

[ Top 10 ] /usr/lib/python3.8/multiprocessing/reduction.py:51: size=23.2 GiB, count=36957, average=659 KiB /usr/lib/python3.8/linecache.py:137: size=508 KiB, count=5133, average=101 B /usr/lib/python3.8/tracemalloc.py:65: size=60.1 KiB, count=962, average=64 B /usr/lib/python3.8/tracemalloc.py:185: size=42.2 KiB, count=900, average=48 B

:1: size=37.7 KiB, count=444, average=87 B :640: size=29.2 KiB, count=388, average=77 B /usr/local/lib/python3.8/dist-packages/kafka/protocol/types.py:193: size=28.4 KiB, count=638, average=46 B /usr/local/lib/python3.8/dist-packages/kafka/metrics/stats/sampled_stat.py:89: size=20.6 KiB, count=416, average=51 B /usr/lib/python3.8/copy.py:76: size=19.2 KiB, count=123, average=160 B /usr/local/lib/python3.8/dist-packages/kafka/cluster.py:281: size=18.4 KiB, count=100, average=188 B When I roll back to 1.1.8 the problem is not reproduced again.