unable to use parfor from within ipython

[GoogleCodeExporter]

Evaluating directly from the ipython prompt

from qutip import *
def func1(x): return x,x**2,x**3
[a,b,c]=parfor(func1,range(10))

aborts with

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 313, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup 
__builtin__.function failed

it's then also not possible to C-c out. Only C-z and SIGKILLing python ends the 
process reliably.

if the same is pasted into python2's prompt i see the expected output (list of 
numpy arrays).

%run example.py
where example.py makes use of parfor, from within ipython also works fine.

What version of the product are you using? On what operating system?

Python 2.7.2
IPython 0.11
qutip r1181

on up-to-date Archlinux.

Thanks for any help.

Original issue reported on code.google.com by cor...@gmail.com on 21 Oct 2011 at 4:10

[GoogleCodeExporter]

It seems that the defining a function in ipython cannot be pickled and sent to 
the thread pool.  This might take some time to figure out.  However, ipython 
includes it's own built in multiprocessing functions for just such cases:

Original comment by nonhermitian on 22 Oct 2011 at 3:14

• Changed state: Accepted

[GoogleCodeExporter]

After a little further digging, it turns out that you cannot pickle functions 
defined interactively (at least without some sneaky ninja tricks).  Therefore, 
this issue is not likely to be solved anytime soon.  We prefer to spend time on 
the scientific code portions.  Instead, it is best to use the built-in 
multiprocessing tools provided by iPython.  However,  maybe someone who knows 
how to do this can come along and provide a fix?

Original comment by nonhermitian on 22 Oct 2011 at 3:35

• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

This works using the developer version of iPython 0.12.  You can download it 
from their website.

Original comment by nonhermitian on 9 Dec 2011 at 2:05

• Changed state: Done