Closed GoogleCodeExporter closed 9 years ago
Update: Rev. 1082 Tests on Windows
-----------------------------------
The attached file show the results of nosetests for rev 1082.
Summary
=======
Ran 730 tests in 128.938s
FAILED (errors=2, failures=10)
Notes
=====
The following directories were excluded from this test run:
pysal/contrib
pysal/core/IOHandlers
I excluded IOHandlers because I haven't figured out yet how to allow Python on
Windows to open more than 512 files at the same time, and that problem
generates a slew of testing errors you can view in the other attachment,
IO-tests.txt.
There is some direction for this limit at
http://www.codeproject.com/KB/system/File_Descriptor_Limit.aspx
Original comment by phil.stp...@gmail.com
on 18 Jan 2012 at 1:14
Attachments:
Update: Revision 1092
Skipped:
pysal/core/IOHandlers -- for reasons stated previously
pysal/spreg/tests -- because they're empty
Added:
tutorial tests run by Sphinx doctest extension
Original comment by phil.stp...@gmail.com
on 19 Jan 2012 at 7:38
Attachments:
Update: Revision 1093
Skipped:
pysal/core/IOHandlers -- for reasons stated previously
pysal/spreg/tests -- because they're empty
Original comment by phil.stp...@gmail.com
on 20 Jan 2012 at 11:01
Attachments:
There seems to be a slight difference in the behavior of pysal FileIO object
and python file object. Namely, python's file object is implemented in C and
uses the "file_dealloc" to close itself when it's no longer referenced [1]. We
are using the __del__ method as an attempt to close the files on
deconstruction. This isn't happening. The __del__ method isn't guaranteed to be
called and is only called when all references to the object are gone [2]. We
probably have some kind of cyclical reference going on.
Refs:
[1]
http://stackoverflow.com/questions/575278/how-does-python-close-files-that-have-
been-gced
[2] http://docs.python.org/reference/datamodel.html#object.__del__
Original comment by schmi...@gmail.com
on 31 Jan 2012 at 12:41
After some further reading...
Garbage Collection in python is used to supplement reference counting. It is
only needed when reference cycles prevent the the reference count from reaching
zero, even though the object is no longer reachable [1].
Ex...
>>> import gc
>>> gc.set_debug(gc.DEBUG_SAVEALL)
>>> class Obj:
... pass
...
>>> a = Obj()
>>> b = Obj()
>>> a.other = b
>>> b.other = a
>>> del a,b
>>> gc.collect()
4
>>> gc.garbage
[<__main__.Obj instance at 0x5089b8>, <__main__.Obj instance at 0x508a08>,
{'other': <__main__.Obj instance at 0x508a08>}, {'other': <__main__.Obj
instance at 0x5089b8>}]
I did a similar debugging with FileIO and it looks like by_row, maybe be
causing a problem. Because by_row is a class which keeps a reference to it's
parent. As such it can only be removed with garbage collection. However,
garbage collection does not destroy object that define __del__ methods, because
it doesn't know what order to call the __del__ methods [2].
Refs:
[1] http://www.kylev.com/2009/11/03/finding-my-first-python-reference-cycle/
[2] http://docs.python.org/library/gc.html#gc.garbage
Original comment by schmi...@gmail.com
on 31 Jan 2012 at 12:51
The following unix command is useful in debugging these problems....
lsof -p PID +r 3
Where PID is the process id of the python interpreter.
this command will list the open files the process is using and repeat every 3
seconds.
Original comment by schmi...@gmail.com
on 31 Jan 2012 at 1:04
Committed a fix/workaround in r1146.
Sending back to Phil to see if this reduces the number of errors in windows.
Original comment by schmi...@gmail.com
on 31 Jan 2012 at 1:07
Results after 1146 attached.
Original comment by phil.stp...@gmail.com
on 31 Jan 2012 at 1:47
Attachments:
As of r1149 I'm seeing the same* errors in OSX and Windows with zero open files
errors remaining.
* I don't have shapely installed on windows, so i'm not seeing those errors on
windows.
Original comment by schmi...@gmail.com
on 31 Jan 2012 at 4:45
Installing Shapely into my Windows development environment. Will generate test
results and post here.
Original comment by phil.stp...@gmail.com
on 31 Jan 2012 at 11:45
Windows test results, r1152 attached. Potentially interesting for spreg and
FileIO developers.
Ran 911 tests in 146.468s
FAILED (errors=9, failures=12)
Original comment by phil.stp...@gmail.com
on 1 Feb 2012 at 12:05
Attachments:
Original issue reported on code.google.com by
schmi...@gmail.com
on 11 Jan 2012 at 11:21Attachments: