get_connections mem leak

[GoogleCodeExporter]

import psutil
import time
from pympler import muppy

def monitorConnections():

    proc_obj_iter = psutil.process_iter()    
    for proc_obj in proc_obj_iter:  
        conn = proc_obj.get_connections()
        del conn

while True:        
    monitorConnections()
    muppy.print_summary()        
    time.sleep(2)  

The function get_conenctions has mem leaks creating tuple references
and the mem is increased at every iteration.(after a while no ram remaining:)

script output:
                       types |   # objects |   total size
============================ | =========== | ============
                        dict |         979 |      2.28 MB
                         str |       19279 |      1.24 MB
is increasing ---->    tuple |        8050 |    615.80 KB
                        code |        2040 |    239.06 KB
                        type |         252 |    220.50 KB
          wrapper_descriptor |        1103 |     77.55 KB
  builtin_function_or_method |         894 |     55.88 KB
                        list |         297 |     39.97 KB
                         int |        1536 |     36.00 KB
                     weakref |         430 |     33.59 KB
           method_descriptor |         413 |     25.81 KB
           member_descriptor |         289 |     18.06 KB
         <class 'abc.ABCMeta |          19 |     16.63 KB
         function (__init__) |         116 |     12.69 KB
                         set |          57 |     12.47 KB

Original issue reported on code.google.com by tiberius...@gmail.com on 18 Nov 2011 at 6:44

[GoogleCodeExporter]

Thanks for the report! Can you clarify which platform and Python version this 
is seen on? The C code for each platform is independent so we'd need to know 
which arch to be looking at. 

Original comment by jlo...@gmail.com on 18 Nov 2011 at 6:48

[GoogleCodeExporter]

windows XP,7 both 32 and 64, python version 2.6 both 32 and 64, psutil verison 
last.
Thank you.

Original comment by tiberius...@gmail.com on 20 Nov 2011 at 1:27

[GoogleCodeExporter]

This should now be fixed as of r1228.
We have a similar problem on OSX.
The patch below should fix it but I don't have an OSX box to test against right 
now.
Jay, could you do it?

Index: psutil/_psutil_osx.c
===================================================================
--- psutil/_psutil_osx.c    (revisione 1224)
+++ psutil/_psutil_osx.c    (copia locale)
@@ -989,6 +989,7 @@
             char lip[200], rip[200];
             char *state;
             int inseq;
+            int _family, _type;

             fd = (int)fdp_pointer->proc_fd;
             family = si.psi.soi_family;
@@ -1002,10 +1003,15 @@
                 continue;

             // apply filters
-            inseq = PySequence_Contains(af_filter, 
PyLong_FromLong((long)family));
+            _family = PyLong_FromLong((long)family);
+            Py_DECREF(_family);
+            inseq = PySequence_Contains(af_filter, _family);
             if (inseq == 0)
                 continue;
-            inseq = PySequence_Contains(type_filter, 
PyLong_FromLong((long)type));
+
+            _type = PyLong_FromLong((long)type);
+            Py_DECREF(_type);
+            inseq = PySequence_Contains(type_filter, _type));
             if (inseq == 0)
                 continue;

Original comment by g.rodola on 20 Nov 2011 at 4:10

• Added labels: Milestone-0.4.1, OpSys-OSX, OpSys-Windows, Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Ok, I got a chance to look at this, noted a few things: 

1) I can't reproduce a tuple leak on OS X even before the changes. List count 
goes up with each iteration however... not sure if this is the actual problem 
we should be looking at? 

2) your patch above has a couple issues:  
2a) _family and _type should be type PyObject* not ints if you're going to pass 
them to PySequence_Contains like that 
2b) I think you want Py_DECREF() called AFTER you call PySequence_Contains() - 
it doesn't make any sense to dereference the variable before you're done using 
it.
2c) I thought we had agreed to avoid the if statement form without brackets? It 
makes future changes more likely to result in a logic bug and/or makes code 
harder to read, and is generally not recommended. 

3) I tried the changes anyway and regardless, I see no difference in behavior 
with the number of tuple objects remaining stable and only the list count 
climbing. 

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:04

[GoogleCodeExporter]

> I can't reproduce a tuple leak on OS X even before the changes. 
> List count goes up with each iteration however... not sure if 
> this is the actual problem we should be looking at? 

Note that for reproducing this issue I haven't used the OP's script. I used 
this one instead:

import socket, psutil
s = socket.socket()
s.bind(('', 0))
s.listen(1)
p = psutil.Process(os.getpid())
while 1:
    p.get_connections()
    print p.get_memory_info()

If the memory info printed on stdout keeps increasing it means there's a leak.
For Windows this was due to 2 factors:
- Py_DECREF wasn't called against the address tuple
- the object returned by PyLong_FromLong((long)type) wasn't Py_DECREFed.

With this two fixes the script above stopped showing increasing values.
Therefore, for starters I'd see what the script above shows.

> 2b) I think you want Py_DECREF() called AFTER you call 
> PySequence_Contains() - it doesn't make any sense to 
> dereference the variable before you're done using it.

Yeah, right. 

> 2c) I thought we had agreed to avoid the if statement 
> form without brackets?

My bad. Feel free to fix it.

Original comment by g.rodola on 23 Nov 2011 at 10:20

[GoogleCodeExporter]

Ok, I tested with your script and the changes, and memory usage is not 
increasing. Looks ok from my end so I've committed as r1229 

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:32

[GoogleCodeExporter]

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:32

• Changed state: FixedInSVN

[GoogleCodeExporter]

Note: I only checked in the OS X side, if you didn't commit changes to Windows 
you'll want to do that as well.

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:33

[GoogleCodeExporter]

You mean the memory wasn't increasing before your change?

Original comment by g.rodola on 23 Nov 2011 at 10:37

[GoogleCodeExporter]

> You mean the memory wasn't increasing before your change?

I didn't test with your script until after I applied the patch changes, but it 
looks good with the patch applied so I've committed it in r1229

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:48

[GoogleCodeExporter]

It was good to figure out whether it afftected OSX as well for the record.
Nevermind... since we weren't DECREFing PyLong_FromLong I assume it almost 
certainly did.

Original comment by g.rodola on 23 Nov 2011 at 10:55

[GoogleCodeExporter]

Just for completeness' sake, I checked with the code reverted to the old 
version and yes, there was a memory leak prior to this patch.

Original comment by jlo...@gmail.com on 23 Nov 2011 at 10:59

[GoogleCodeExporter]

Original comment by g.rodola on 14 Dec 2011 at 11:18

• Changed state: Fixed

[GoogleCodeExporter]

This is now fixed in 0.4.1 version.

Original comment by g.rodola on 14 Dec 2011 at 11:51

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Updated csets after the SVN -> Mercurial migration:
r1228 == revision 0adfedd760c3
r1229 == revision abd221389d83

Original comment by g.rodola on 2 Mar 2013 at 12:05