mmckerns
commented
1 year ago Sorry for the slow reply. I'm seeing the same issue as you...
Results from the interpreter:
Python 3.7.15 (default, Oct 12 2022, 04:11:53)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos
>>> p = pathos.pools.ParallelPool()
>>> def fgk(x):
... z = x**2
... y = '1' + z
... return y
...
>>> p.map(fgk, [1,2])
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
[None, None]
>>>
>>> pathos.__version__
'0.3.0.dev0'
>>> import ppft
>>> ppft.__version__
'1.7.6.6.dev0'
>>> import dill
>>> dill.__version__
'0.3.6.dev0'
>>>
>>> try:
... p.map(fgk, [1,2])
... except Exception:
... print('Error caught!')
...
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
[None, None]
>>>
>>> def fgt(x):
... z = x**2
... try:
... y = '1' + z
... except Exception:
... y = z
... return y
...
>>> p.map(fgt, [1,2])
[1, 4]Results are no different when running from a file. And when running your code:
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!I also see that the errors are raised, but are not caught. Note that I'm not seeing the same issue you are with the string being printed oddly. Let me know what OS and versions of the code you are using.
Hmm. If I remember correctly ParallelPool doesn't have the same ability to capture a traceback from the worker process and return it to the calling process. I'll take this as a feature request.
zhuchcn
Thanks for making this great library! I have a question regarding to how exception handling could be done with
ParallelPool. UsingProcessPool, errors can be caught as normal. However withParallelPool, the error couldn't be caught likeProcessPoolsee below. It seems like the error of all three worker processes are printed out to stderr directly, but none of then are caught.We also noticed that some binary strings are printed out with the error messages. There is a "%" symbol in the end of the example below, but in real life, we got something like this:
I'm wondering if the
ParallelPoolhas its own way of error handling and please point me to it if it is already documented!