internal IO function(s) not closing files

[GoogleCodeExporter]

It appears there are some remaining places where pysal is not closing files 
once it's done using them.  When using queen_from_shapefile(), I am not able to 
build 359 W objects in one script (note that I am only holding one W in RAM at 
any one time).  Below is some output.  

----------------------------
Traceback (most recent call last):
...
  File ".../pysal/trunk/pysal/weights/user.py", line 55, in queen_from_shapefile
  File ".../pysal/trunk/pysal/core/IOHandlers/pyShpIO.py", line 64, in __init__
  File ".../pysal/trunk/pysal/core/IOHandlers/pyShpIO.py", line 71, in __open
  File ".../pysal/trunk/pysal/core/util/shapefile.py", line 134, in __init__
  File ".../pysal/trunk/pysal/core/util/shapefile.py", line 162, in _open_shp_file
  File ".../pysal/trunk/pysal/core/util/shapefile.py", line 330, in __init__
  File ".../pysal/trunk/pysal/core/util/shapefile.py", line 350, in _open_shx_file
IOError: [Errno 24] Too many open files: '.../42140.shx'

Original issue reported on code.google.com by dfo...@gmail.com on 17 May 2011 at 7:58

[GoogleCodeExporter]

I double checked the code in pysal and it appears that files are being closed. 
Are you opening any other files in your script?  Try using the command line 
util `lsof` to help debug the problem.

Original comment by schmi...@gmail.com on 17 May 2011 at 9:29

[GoogleCodeExporter]

My test script opened many file types.  It turns out the culprit is the DBF 
reader... sorry for implicating the SHP reader.  `lsof` is a great tool.  The 
problem can be replicated with:

import pysal
for i in range(300):
    dbf = pysal.open('my_data.dbf', 'r')
    dbf.close()

Original comment by dfo...@gmail.com on 17 May 2011 at 10:31

[GoogleCodeExporter]

Issue was fixed in r905.  The DBF handler was indeed leaving files open when 
opened in 'r' mode.

Problem should be fixed.  Please re-open this issue if you have any more 
problems.

Original comment by schmi...@gmail.com on 18 May 2011 at 2:49

• Changed state: Fixed
• Added labels: OpSys-All, Milestone-1.2, Component-Logic, Performance