Memory leak with C-extension?

[alvinchow86]

I suspect there might be a memory leak with the C-extension verion of Proxy. I have some code (in a Django app) which creates a large number of Proxy objects that point to simple Python objects (larg-ish strings). These objects are placed into a list which is periodically cleared (after they are "consumed" by application code), so they should be garbage-allocated. However, I noticed that my Python process started growing continually (from about 300mb to 1gb to 2gb, and so on).

I then switched to the "simple" and the "slots" Python implementations and the memory issues went away (my process memory usage would fluctuate up and down but remain pretty stable).

Would be happy to provide more details or try to create a test case! (It is also possible I'm doing something wrong in my code)

[ionelmc]

A test case would be ideal.

Also, knowing what operations you use on the objects could be useful (assuming one of the proxy methods is leaky).

There's one particular thing about the cext implementation - I'm not sure CPython's cycle detector works with it. Are you sure you don't have reference cycles?

[ionelmc]

Note to self: missing handling for self->factory in https://github.com/ionelmc/python-lazy-object-proxy/blob/master/src/lazy_object_proxy/cext.c#L114-L131 might be the cause.

[alvinchow86]

I don't believe I have reference cycles, the object is read and assigned to another object briefly, but that other object is destroyed and no longer used. There are several different types of objects, the operations are serializing to JSON and comparison (==), but I have to investigate more.

I will try to set up a test case and that should make clarify everything more. I'll also try what you mentioned with self->factory and see if it improves anything.

Thanks for looking into this!

[alvinchow86]

FWIW I tried adding PyVisit(self->factory) (https://github.com/ionelmc/python-lazy-object-proxy/compare/master...chillbear:memory-leak-fix) and that seemed to help with the memory leak (not sure if this was the right thing to do, just trying stuff)

[ionelmc]

Looks right, open up a pr with that. On Wed 1 Jul 2015 at 03:20 Alvin Chow notifications@github.com wrote:

FWIW I tried adding PyVisit(self->factory) ( master...chillbear:memory-leak-fix https://github.com/ionelmc/python-lazy-object-proxy/compare/master...chillbear:memory-leak-fix) and that seemed to help with the memory leak (not sure if this was the right thing to do, just trying stuff)

— Reply to this email directly or view it on GitHub https://github.com/ionelmc/python-lazy-object-proxy/issues/6#issuecomment-117377384 .

[alvinchow86]

FYI after the fix I was still noticing some memory-leak/growth with my same test on the C-extension, so still using the Python implementation for now.

[ionelmc]

Oh damn. Could you make an example that reproduces the issue? (like some minimal django app/project that has the problem).

Also, if that's very hard, you could make some graph of the objects, like this http://blog.ionelmc.ro/2013/06/05/python-debugging-tools/#memory-leaks. I'll take a look if you show me some data or graphs.

Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Wed, Jul 22, 2015 at 2:53 AM, Alvin Chow notifications@github.com wrote:

FYI after the fix I was still noticing some memory-leak/growth with my same test on the C-extension, so still using the Python implementation for now.

— Reply to this email directly or view it on GitHub https://github.com/ionelmc/python-lazy-object-proxy/issues/6#issuecomment-123520741 .

[alvinchow86]

Thanks, I'll try to get you something over the next week if I can. Definitely understand that it helps to have some examples / test cases before trying to debug =)