ROD problems on W objects when passing them to multi-core functions

[GoogleCodeExporter]

When passing a W object into a function to be applied in several cores using 
multiprocessing, Read-Only Dictionaries cause the process to fail giving the 
following error:

padma:Desktop dani$ python multi_rod.py 
Process PoolWorker-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process PoolWorker-2:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    self.run()
    task = get()
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
  File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
  File "/Users/dani/repos/pysal/pysal/common.py", line 25, in __setitem__
    return recv()
  File "/Users/dani/repos/pysal/pysal/common.py", line 25, in __setitem__
    raise TypeError, "'Read Only Dictionary (ROD)' object does not support item assignment"
    raise TypeError, "'Read Only Dictionary (ROD)' object does not support item assignment"
TypeError: 'Read Only Dictionary (ROD)' object does not support item assignment
TypeError: 'Read Only Dictionary (ROD)' object does not support item assignment

See attached script as example.

Original issue reported on code.google.com by dreamessence on 23 Feb 2011 at 10:18

Attachments:

• multi_rod.py

[GoogleCodeExporter]

or, one could say the multiprocessing is trying to set items in the ROD that 
raise the TypeError. We'd have to know more about what multiprocessing is doing 
here. it isn't obvious to me why multiprocessing results in the W having its 
values set after instantiation?

Original comment by sjsrey on 23 Feb 2011 at 11:34

[GoogleCodeExporter]

Changing to a feature request, as this isn't a bug in the ROD. 

Original comment by sjsrey on 23 Feb 2011 at 11:35

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

A work around is to build the W inside the get_lag() function in the example 
script.

Original comment by dfo...@gmail.com on 23 Feb 2011 at 11:54

[GoogleCodeExporter]

Right, I expressed myself not correctly, I don't think it's a bug in PySAL per 
se. My sense is that multiprocessing is trying to reset the RODs for some 
reason (maybe the way it's structured it resets them to the same values for 
some reason) and then the RODs raise the TypeError. I've tried commenting out 
line 25 in pysal/common.py to allow for the changes to be done and it runs. At 
the end I compare the resulting W with the original one and the full arrays are 
equal (using binary contiguity though, so it might be too simple of an example).

Maybe the RODs could be set to check when they are going to be changed, if 
they're going to be replaced by the same values, it could be allowed.

Original comment by dreamessence on 23 Feb 2011 at 11:56

[GoogleCodeExporter]

python's copy.copy method chokes on pysal's ROD.  I believe this is what 
multiprocessing was doing.

{{{>>> copy.copy(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/copy.py", line 95, in copy
    return _reconstruct(x, rv, 0)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/copy.py", line 335, in _reconstruct
    y[key] = value
  File "/Users/charlie/Documents/repos/pysal/trunk/pysal/common.py", line 25, in __setitem__
    raise TypeError, "'Read Only Dictionary (ROD)' object does not support item assignment"
TypeError: 'Read Only Dictionary (ROD)' object does not support item assignment
}}}

I added a "__copy__" method to ROD in rev. 864.  shallow copy now work.  If mp 
is using deepcopy, we'll need to implement a deepcopy.

Dani, can you try running your tests again?

Original comment by schmi...@gmail.com on 14 Apr 2011 at 12:14

• Added labels: Milestone-1.2

[GoogleCodeExporter]

This change does not seem to fix the problem.  It's not clear how 
multiprocessing is trying to copy the ROD.

Original comment by schmi...@gmail.com on 19 May 2011 at 6:09

[GoogleCodeExporter]

One workaround is to just pass the W.sparse attribute to multiprocessing. 

Original comment by dfo...@gmail.com on 19 May 2011 at 6:13

• Changed state: WontFix