Memory leak in a long running application

[BilalMAddam]

Hello,

I'm using openhtf as the core module tester in my application. It consists of a long running process. I'm encountering a huge memory leak. To validate my speculation, I took the hello world example and ran it in a loop which is similar of what I need in my application (create a thread per test and dispatch it...). The memory usage is indeed increasing. Here's a snapshot of the memory after a couple of iterations of the hello world example.

*** Logs (see the SummaryTracker print_diff at the end of each 100 iterations) hello_world_mm_leak.log

*** Script (as a text file to be uploaded) hello_world_openhtf.py.txt

I'd like to add that by using Processes, I've no longer encountered a memory leak. However, the use of them is a bit overkill in my application and creates other ones.

Thoughts?

[arsharma1]

I've been investigating this and see that the garbage collector shows a list object in each phase run that looks like: z = [] y = [z, z, 'phase_result'] z += [y, y, None]

I'm trying to determine where this recursive structure is coming from.

[arsharma1]

Added #861 that significantly reduces the leaks. I still see some, but the grown is much smaller.

Another question: why does your example code create a new Test object each time? You can just construct it once and call its execute function repeatedly (as long as you don't try to call it simultaneously).

[BilalMAddam]

What would be the downside of reconstructing the test? Is it recommended to only create it once? Of course, I could add a cache to my test factory and look it up when I need to run the same test over and over.

[arsharma1]

Some of the other leaks are coming from the JSONEncoder due to references with closures that are not well handled. I will research more into resolving that part.

There's one other leak that's probably coming from something similar inside our code. I'll try to search for those as well.

[BilalMAddam]

Plz do keep me posted @arsharma1

[arsharma1]

862 fixes the remaining issue. In local testing, I don't see any memory leaks in Python3 and just fluctuations in Python2.