Bug in Pool.imap?

[GoogleCodeExporter]

When I run the following simple code:
from multiprocessing import Pool

f = lambda x: x*x
p=Pool()
res = p.imap(f,range(10))

I am getting the following error both in the backport to python 2.5 and in
python 3.0:

Exception in thread Thread-1:                         
Traceback (most recent call last):                                        

  File "/usr/lib/python3.0/threading.py", line 507, in _bootstrap_inner   

    self.run()                                                            

  File "/usr/lib/python3.0/threading.py", line 462, in run                

    self._target(*self._args, **self._kwargs)                             

  File "/usr/lib/python3.0/multiprocessing/pool.py", line 225, in
_handle_tasks                  
    put(task)                                                             

  File "/usr/lib/python3.0/pickle.py", line 1317, in dumps                

    Pickler(f, protocol).dump(obj)                                        

_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup
builtins.function failed

I am using ubuntu intrepid on a centrino core2 duo based system

Original issue reported on code.google.com by fccoelho on 7 Nov 2008 at 11:31

[GoogleCodeExporter]

It has nothing to do with the multiprocessing.  You can't pickle lambdas, at 
least not in that context:

>>> import pickle
>>> f = lambda x: x*x
>>> pickle.dumps(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/skip/local/lib/python2.7/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/Users/skip/local/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/Users/skip/local/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/skip/local/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function <lambda> at 0x6683f0>: it's not found as 
__main__.<lambda>

However, if you define f using the def statement it works:

>>> def f(x):
...   return x*x
... 
>>> pickle.dumps(f)
'c__main__\nf\np0\n.'

Original comment by skip.mon...@gmail.com on 8 Nov 2008 at 12:34

• Changed state: Invalid
• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

You are right, but if you define the function, it will run on Python3.0 but 
will hang
on python 2.5 (at least in my machine). Here is the code I tested:

from multiprocessing import Pool
#from processing import Pool

def f (x):
    return x*x
p=Pool(processes=2)
p.map(f,range(1000000))
#print (res)
#res = p.apply_async(f,2)

Original comment by fccoelho on 8 Nov 2008 at 2:38

[GoogleCodeExporter]

Actually you could use a hack to pickle lambda functions. The only thing you are
interested in a function is actually its func_code object, which is 
marshallable. So
you can dump lambda functions to a string, see the attached file.

Original comment by Vinzent.Steinberg@gmail.com on 31 Jan 2010 at 3:39

Attachments:

• serial_func.py