search

tkuester / taky

A simple python TAK server
MIT License
188 stars 43 forks source link

Memory Leak #40

Closed tkuester closed 2 years ago

tkuester commented 2 years ago

Many users have observed a potential memory leak in the server. It's currently suspected that this is due to a bug with clients connecting and disconnecting, and the server not cleaning up the old client object.

I'm unable to address this issue right now, but hopefully will be able to find some time in October. However, this is the highest priority issue. If anyone is able to help track this down, it would be wonderful!

tkuester commented 2 years ago

I did some tests to try and narrow the scope of what's wrong, and it seems that this is a known issue with LXML. There may be some hacks that can be done, like restarting the LXML parser -- but this is a little hacky, and I'm not terribly fond of it.

tkuester commented 2 years ago

It appears that LXML is known to have a memory leak, and that there is no simple way to work around this.

This is greatly upsetting to mark as "wontfix", but I don't see a solution to this now, outside of writing a new XML parser, or creating a subprocess to handle packets. (Eww! Things are already bad enough with how I'm handling XML...)

1 - https://www.reddit.com/r/Python/comments/j0gl8t/psa_pythonlxml_memory_leaks_and_a_solution/

tkuester commented 2 years ago

As a note, though I've marked this issue as "wontfix", it still keeps me awake at night! ;)

I tried replacing LXML with Python's built in XML parser, and the memory leak was orders of magnitude worse.

There's still the option of writing a SAX based parser using expat, but the complexity and overhead seems daunting.

One last option (according to the Reddit thread) is to spawn the XML parser in a multiprocess, and respawn it once the memory usage goes too high... but this is a hack upon a hack.