(Windows7-64bits) get_memory_info fails when called toooo many times

[GoogleCodeExporter]

It seams that there is a limit to the number of times that we can call 
get_memory_info for a process. This causes problems when I need to
monitor the memory usage of a very long process. 

Run the attached file on a windows 7 64-bit machine (python 2.7, psutil 0.4.0) 
to replicate the issue. The program starts an infinate loop that asks for the 
current process memory. If left alone it should run forever but it fails with:

WindowsError: [Error 1450] Insufficient system resources exist to complete the 
requested service

The problem happens each time when the function has been called ~16711544 
times. 

Thanks,
Ioannis

Original issue reported on code.google.com by ITzia...@gmail.com on 8 Dec 2011 at 4:55

Attachments:

• issue.py

[GoogleCodeExporter]

That sounds like it may be a memory leak issue. Are you able to run the 
test_memory_leaks.py script on your system? (It's in psutil test/ directory if 
you download the source). 

Original comment by jlo...@gmail.com on 8 Dec 2011 at 5:11

• Added labels: 64bit, OpSys-Windows7

[GoogleCodeExporter]

Running the memory leak test results in three failures, but not at the 
get_memory_info (see traceback). I think that the problem is that every time
a new process handle is opened in _psutil_mswindows.c. Maybe this should be 
done once at the object creation. Furthermore, microsoft suggests that the 
process handle has been opened with PROCESS_QUERY_INFORMATION | PROCESS_VM_READ
(http://msdn.microsoft.com/en-us/library/windows/desktop/ms682050(v=vs.85).aspx)
this might be also something to look at.

--------------
Test Traceback
--------------

test_cmdline (__main__.TestProcessObjectLeaks) ... ok
test_create_time (__main__.TestProcessObjectLeaks) ... ok
test_get_connections (__main__.TestProcessObjectLeaks) ... FAIL
test_get_cpu_times (__main__.TestProcessObjectLeaks) ... ok
test_get_memory_info (__main__.TestProcessObjectLeaks) ... ok
test_get_num_threads (__main__.TestProcessObjectLeaks) ... ok
test_get_open_files (__main__.TestProcessObjectLeaks) ... ok
test_get_threads (__main__.TestProcessObjectLeaks) ... ok
test_getcwd (__main__.TestProcessObjectLeaks) ... ok
test_gids (__main__.TestProcessObjectLeaks) ... skipped-ok
test_is_running (__main__.TestProcessObjectLeaks) ... ok
test_name (__main__.TestProcessObjectLeaks) ... ok
test_ppid (__main__.TestProcessObjectLeaks) ... ok
test_resume (__main__.TestProcessObjectLeaks) ... ok
test_status (__main__.TestProcessObjectLeaks) ... ok
test_terminal (__main__.TestProcessObjectLeaks) ... skipped-ok
test_uids (__main__.TestProcessObjectLeaks) ... skipped-ok
test_username (__main__.TestProcessObjectLeaks) ... FAIL
test_cpu_times (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_io_counters (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_partitions (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_usage (__main__.TestModuleFunctionsLeaks) ... ok
test_get_pid_list (__main__.TestModuleFunctionsLeaks) ... ok
test_network_io_counters (__main__.TestModuleFunctionsLeaks) ... FAIL
test_per_cpu_times (__main__.TestModuleFunctionsLeaks) ... ok
test_phymem_usage (__main__.TestModuleFunctionsLeaks) ... ok
test_pid_exists (__main__.TestModuleFunctionsLeaks) ... ok
test_process_iter (__main__.TestModuleFunctionsLeaks) ... ok
test_virtmem_usage (__main__.TestModuleFunctionsLeaks) ... ok

======================================================================
FAIL: test_get_connections (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ionnis\Projects\psutil-0.4.0\test\test_psutil.py", line 150, in inner
    return fun(self, *args, **kwargs)
  File ".\test_memory_leaks.py", line 158, in test_get_connections
    self.execute('get_connections')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=15777792, rss3=16179200, difference=401408

======================================================================
FAIL: test_username (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ionnis\Projects\psutil-0.4.0\test\test_psutil.py", line 150, in inner
    return fun(self, *args, **kwargs)
  File ".\test_memory_leaks.py", line 120, in test_username
    self.execute('username')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=17195008, rss3=17219584, difference=24576

======================================================================
FAIL: test_network_io_counters (__main__.TestModuleFunctionsLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".\test_memory_leaks.py", line 202, in test_network_io_counters
    self.execute('network_io_counters')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=17641472, rss3=17666048, difference=24576

----------------------------------------------------------------------
Ran 29 tests in 466.125s

FAILED (failures=3)

Original comment by ioann...@enthought.com on 8 Dec 2011 at 9:16

[GoogleCodeExporter]

Does the memory information get returned properly? 

I can't see any obvious reason a handle leak would be occurring. We close the 
handle on all code paths in the memory function itself. 

Original comment by jlo...@gmail.com on 8 Dec 2011 at 9:26

[GoogleCodeExporter]

I cannot figure it out either, I have tried using pywin32 and it has no problem 
going
upto 2**25 call times. However, I am using it with PROCESS_QUERY_INFORMATION | 
PROCESS_VM_READ.

Original comment by ioann...@enthought.com on 8 Dec 2011 at 10:23

[GoogleCodeExporter]

Well, I ran the test script for over an hour on my system and can't get it to 
error out. However, I'm not on Windows 7 64bit so that may be a factor. 
Giampaolo might be able to test, I think he has a Win 7 system. 

The flags used when the handle is opened wouldn't have an effect on this issue, 
but in any case, the handle used by psutil for get_memory_info() is already 
being opened with PROCESS_QUERY_INFORMATION | PROCESS_VM_READ 
(handle_from_pid() function in psutil/arch/mswindows/process_info.c) 

There shouldn't be any functional difference but is it possible for you to try 
with the latest SVN version just in case? 

Original comment by jlo...@gmail.com on 8 Dec 2011 at 10:47

[GoogleCodeExporter]

Looking at the SVN version I can see that there is an additional CloseHandle 
call
which is not present in the 0.4.0 source. I will try to compile the updated 
source, but I think that it should work fine.

Original comment by ioann...@enthought.com on 8 Dec 2011 at 11:34

[GoogleCodeExporter]

Hrm, my fault then - I thought that change had made it into 0.4.0 but 
apparently not. That would certainly account for the issue if there's a handle 
leak since the error code indicates running out of file descriptors/handles. 

Original comment by jlo...@gmail.com on 9 Dec 2011 at 12:02

[GoogleCodeExporter]

There was an unclosed handle indeed, fixed in r1219 by David who didn't file an 
issue or updated HISTORY.

Original comment by g.rodola on 9 Dec 2011 at 7:41

• Changed state: FixedInSVN
• Added labels: Milestone-0.4.1

[GoogleCodeExporter]

Original comment by g.rodola on 14 Dec 2011 at 11:17

• Added labels: OpSys-Windows, Priority-High
• Removed labels: OpSys-Windows7, Priority-Medium

[GoogleCodeExporter]

Original comment by g.rodola on 14 Dec 2011 at 11:18

• Changed state: Fixed

[GoogleCodeExporter]

This is now fixed in 0.4.1 version.

Original comment by g.rodola on 14 Dec 2011 at 11:51

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Updated csets after the SVN -> Mercurial migration:
r1219 == revision d6716872097a

Original comment by g.rodola on 2 Mar 2013 at 12:06