uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

How to catch error information of sub-processing when using pathos.multiprocessing.Pool? #207

Closed YanxiLiuuuu closed 1 year ago

YanxiLiuuuu commented 3 years ago

Hello,

I am working on a project and I am using pathos to run multiprocessing sample. I try to catch the detailed error/ coding bug information from sub-processing for debugging. For example, I hope to know the detailed location of the wrong coding, in which python file and which line.  

I am using logging and traceback to get the information of the wrong coding, but it does not show detailed error information of sub-processing. And, the python version I am using is 2.7. I am looking forward for your reply. Thanks for your time!

Best, Yanxi

mmckerns commented 1 year ago

Sorry your question was a bit ignored. 2.7 is no longer supported, so I'm responding with 3.x:

Python 3.7.15 (default, Oct 12 2022, 04:11:53) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo(x):
...   x += 2
...   z = x**2
...   y = '1' + z
...   return y
... 
>>> import pathos.pools as pp
>>> p_ = pp.ProcessPool()
>>> p_.map(foo, [1,2])
multiprocess.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/mmckerns/lib/python3.7/site-packages/pathos/helpers/mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "<stdin>", line 4, in foo
TypeError: can only concatenate str (not "int") to str
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/pathos/multiprocessing.py", line 135, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
TypeError: can only concatenate str (not "int") to str
>>> 

Please reopen if this doesn't answer your question.