search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.39k stars 89 forks source link

cannot attach debugger using pycharm #113

Open ni-tterry opened 7 years ago

ni-tterry commented 7 years ago

Hey folks! Thanks for your work on this extensive multiprocessing fork.

I am looking at pathos to circumvent some pickling problems using traditional multiprocessing. However, it seems like I can't attach a debugger anytime after a ParallelPool is instantiated:

Connected to pydev debugger (build 143.2370)
INFO|pp|Creating server instance (pp-1.6.4.7.1)
INFO|pp|Running on Python 2.7.10 darwin
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 2411, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1802, in run
    launch(file, globals, locals)  # execute the script
  File "/Users/tterry/iZotope/dsptester/dsptester/run.py", line 14, in <module>
    run()
  File "/Users/tterry/iZotope/dsptester/dsptester/run.py", line 8, in run
    testrunner = DSPTestRunner()
  File "/Users/tterry/iZotope/dsptester/dsptester/dsptestrunner.py", line 38, in __init__
    self.pool = Pool()
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pathos/parallel.py", line 183, in __init__
    _pool = self._serve(nodes=ncpus, servers=servers)
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pathos/parallel.py", line 203, in _serve
    _pool = pp.Server(ppservers=servers)
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pp.py", line 357, in __init__
    self.set_ncpus(ncpus)
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pp.py", line 521, in set_ncpus
    range(ncpus - len(self.__workers))])
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pp.py", line 155, in __init__
    self.start()
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pp.py", line 167, in start
    self.pid = int(self.t.receive())
  File "/Users/tterry/Envs/dsptester-env/lib/python2.7/site-packages/pptransport.py", line 144, in receive
    raise RuntimeError("Communication pipe read error")
RuntimeError: Communication pipe read error

This happens any time I hit a breakpoint after the ParallelPool is created. It does not happen with pathos.multiprocessing.Pool.

mmckerns commented 7 years ago

Ok, this is interesting. I expect that what's happening is that the breakpoint somehow makes the object unserializable -- just as a guess. ppft uses dill.source for serialization (not pickling, but extracting the relevant source code from the object, and passing that).

Could you post a small bit of code that produces the issue? This way, I can try it. Also versions of the codes you are using is helpful information.

lsmithiii commented 7 years ago

Hi, I'm having the same issue. I've had a previous problem with parallel pool where I was getting different results for my parallel code than I was getting for single processing code (my code is a Class where use a univariate spline interpolation to transform/smooth a noisey data set). Anwyay, I tried to debug in PyCharms and got the below output after my breakpoints for ParallelPool. I'm not sure if this is the reason as to why I'm getting the different outputs for parallel and single processing, as I have not been able to find one, but maybe this has something to do with it. Nonetheless I'm a bit lost and have found little on the web that explains this Communication pipe read error. Any help or advice as how to I should proceed would be greatly appreciated. Here is my PyCharm's debugger output,

Connected to pydev debugger (build 172.3317.103) Traceback (most recent call last): File "C:\Users\lloyd.smith\PyCharm Community Edition 2017.2\helpers\pydev\pydevd.py", line 1596, in globals = debugger.run(setup['file'], None, None, is_module) File "C:\Users\lloyd.smith\PyCharm Community Edition 2017.2\helpers\pydev\pydevd.py", line 1023, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:/Users/lloyd.smith/2ke.py", line 710, in parallel_pls_compute() File "C:/Users/lloyd.smith/2ke.py", line 597, in parallel_pls_compute tri_spli = sp.para_train(col_lis) File "C:/Users/lloyd.smith/2ke.py", line 516, in para_train pool = Pool(8) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pathos\parallel.py", line 184, in init _pool = self._serve(nodes=ncpus, servers=servers) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pathos\parallel.py", line 204, in _serve _pool = pp.Server(ppservers=servers) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pp.py", line 357, in init self.set_ncpus(ncpus) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pp.py", line 521, in set_ncpus range(ncpus - len(self.workers))]) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pp.py", line 155, in init__ self.start() File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pp.py", line 167, in start self.pid = int(self.t.receive()) File "C:\Users\lloyd.smith\AppData\Local\Continuum\Anaconda2\lib\site-packages\pptransport.py", line 144, in receive raise RuntimeError("Communication pipe read error") RuntimeError: Communication pipe read error

Process finished with exit code 1

lsmithiii commented 7 years ago

I think the issue is with ppft and pycharm. I changed my code so it used ProcessPool from pathos.multiprocessing and the 'communication pipe read erro'r stopped occurring. After making this change, I'm now getting a cPickle.PicklingError. Fortunately, I switched over to a Mac Os operating system and the code runs just fine with no errors.