Unittest Failures on Windows

[GoogleCodeExporter]

"""Devs,

I've attached some files with nosetest results for PySAL on Windows XP. 

Mostly we're still having the annoying problem of files not being closed :
WindowsError: [Error 32] The process cannot access the file because it is being 
used by another process:

The CG module is throwing one error that appears to be valid. 

For the most part, we have lots of floating point precision discrepancies.""" 
-- Phil

Original issue reported on code.google.com by schmi...@gmail.com on 11 Jan 2012 at 11:21

Attachments:

• cg.txt
• IO.txt

[GoogleCodeExporter]

Update: Rev. 1082 Tests on Windows
-----------------------------------
The attached file show the results of nosetests for rev 1082. 

Summary
=======
Ran 730 tests in 128.938s

FAILED (errors=2, failures=10)

Notes
=====
The following directories were excluded from this test run:

pysal/contrib
pysal/core/IOHandlers

I excluded IOHandlers because I haven't figured out yet how to allow Python on 
Windows to open more than 512 files at the same time, and that problem 
generates a slew of testing errors you can view in the other attachment, 
IO-tests.txt.

There is some direction for this limit at 
http://www.codeproject.com/KB/system/File_Descriptor_Limit.aspx 

Original comment by phil.stp...@gmail.com on 18 Jan 2012 at 1:14

Attachments:

• testoutput.txt
• IO-tests.txt

[GoogleCodeExporter]

Update: Revision 1092

Skipped:
pysal/core/IOHandlers  -- for reasons stated previously
pysal/spreg/tests  -- because they're empty

Added:
tutorial tests run by Sphinx doctest extension

Original comment by phil.stp...@gmail.com on 19 Jan 2012 at 7:38

Attachments:

• pysaltests.txt

[GoogleCodeExporter]

Update: Revision 1093

Skipped:
pysal/core/IOHandlers  -- for reasons stated previously
pysal/spreg/tests  -- because they're empty

Original comment by phil.stp...@gmail.com on 20 Jan 2012 at 11:01

Attachments:

• pysaltests.txt

[GoogleCodeExporter]

There seems to be a slight difference in the behavior of pysal FileIO object 
and python file object.  Namely, python's file object is implemented in C and 
uses the "file_dealloc" to close itself when it's no longer referenced [1]. We 
are using the __del__ method as an attempt to close the files on 
deconstruction. This isn't happening. The __del__ method isn't guaranteed to be 
called and is only called when all references to the object are gone [2].  We 
probably have some kind of cyclical reference going on.

Refs: 
[1] 
http://stackoverflow.com/questions/575278/how-does-python-close-files-that-have-
been-gced
[2] http://docs.python.org/reference/datamodel.html#object.__del__

Original comment by schmi...@gmail.com on 31 Jan 2012 at 12:41

[GoogleCodeExporter]

After some further reading...

Garbage Collection in python is used to supplement reference counting. It is 
only needed when reference cycles prevent the the reference count from reaching 
zero, even though the object is no longer reachable [1].

Ex...
>>> import gc
>>> gc.set_debug(gc.DEBUG_SAVEALL)
>>> class Obj:
...   pass
... 
>>> a = Obj()
>>> b = Obj()
>>> a.other = b
>>> b.other = a
>>> del a,b
>>> gc.collect()
4
>>> gc.garbage
[<__main__.Obj instance at 0x5089b8>, <__main__.Obj instance at 0x508a08>, 
{'other': <__main__.Obj instance at 0x508a08>}, {'other': <__main__.Obj 
instance at 0x5089b8>}]

I did a similar debugging with FileIO and it looks like by_row, maybe be 
causing a problem. Because by_row is a class which keeps a reference to it's 
parent.  As such it can only be removed with garbage collection.  However, 
garbage collection does not destroy object that define __del__ methods, because 
it doesn't know what order to call the __del__ methods [2].

Refs:
[1]  http://www.kylev.com/2009/11/03/finding-my-first-python-reference-cycle/
[2]  http://docs.python.org/library/gc.html#gc.garbage

Original comment by schmi...@gmail.com on 31 Jan 2012 at 12:51

[GoogleCodeExporter]

The following unix command is useful in debugging these problems....

lsof -p PID +r 3

Where PID is the process id of the python interpreter.

this command will list the open files the process is using and repeat every 3 
seconds.

Original comment by schmi...@gmail.com on 31 Jan 2012 at 1:04

[GoogleCodeExporter]

Committed a fix/workaround in r1146.

Sending back to Phil to see if this reduces the number of errors in windows.

Original comment by schmi...@gmail.com on 31 Jan 2012 at 1:07

[GoogleCodeExporter]

Results after 1146 attached. 

Original comment by phil.stp...@gmail.com on 31 Jan 2012 at 1:47

Attachments:

• windows_tests.txt

[GoogleCodeExporter]

As of r1149 I'm seeing the same* errors in OSX and Windows with zero open files 
errors remaining.

* I don't have shapely installed on windows, so i'm not seeing those errors on 
windows.

Original comment by schmi...@gmail.com on 31 Jan 2012 at 4:45

• Changed state: Fixed

[GoogleCodeExporter]

Installing Shapely into my Windows development environment. Will generate test 
results and post here.

Original comment by phil.stp...@gmail.com on 31 Jan 2012 at 11:45

[GoogleCodeExporter]

Windows test results, r1152 attached. Potentially interesting for spreg and 
FileIO developers. 

Ran 911 tests in 146.468s

FAILED (errors=9, failures=12)

Original comment by phil.stp...@gmail.com on 1 Feb 2012 at 12:05

Attachments:

• windows_tests.txt