Open zhuchcn opened 2 years ago
Sorry for the slow reply. I'm seeing the same issue as you...
Results from the interpreter:
Python 3.7.15 (default, Oct 12 2022, 04:11:53)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathos
>>> p = pathos.pools.ParallelPool()
>>> def fgk(x):
... z = x**2
... y = '1' + z
... return y
...
>>> p.map(fgk, [1,2])
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
[None, None]
>>>
>>> pathos.__version__
'0.3.0.dev0'
>>> import ppft
>>> ppft.__version__
'1.7.6.6.dev0'
>>> import dill
>>> dill.__version__
'0.3.6.dev0'
>>>
>>> try:
... p.map(fgk, [1,2])
... except Exception:
... print('Error caught!')
...
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 3, in fgk
TypeError: can only concatenate str (not "int") to str
[None, None]
>>>
>>> def fgt(x):
... z = x**2
... try:
... y = '1' + z
... except Exception:
... y = z
... return y
...
>>> p.map(fgt, [1,2])
[1, 4]
Results are no different when running from a file. And when running your code:
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
An error has occured during the function execution
Traceback (most recent call last):
File "/Users/mmckerns/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
I also see that the errors are raised, but are not caught. Note that I'm not seeing the same issue you are with the string being printed oddly. Let me know what OS and versions of the code you are using.
Hmm. If I remember correctly ParallelPool
doesn't have the same ability to capture a traceback from the worker process and return it to the calling process. I'll take this as a feature request.
I tried several things.
docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) python:3.7.15 /bin/bash -c "pip install pathos==0.2.8; python test.py"
.Not very sure what's causing it but this might be relevant to the OS.
is there a reason you are using pathos
0.2.8 instead of 0.3.0? Can you try against a fresh release of pathos
(and it's dependencies) instead of 0.2.8? Are you using that version everywhere (i.e. it'd be helpful to know the versions of pathos
, ppft
, and dill
you are using).
0.2.8 is the version that I used to use when I saw the issue with the binary symbols. Here is what I just tried:
docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) python:3.7.15 /bin/bash -c "pip install pathos==0.3.0; pip freeze; python test.py" > test.out
Below is what got written into test.out
, minus the output for pip.
Installing collected packages: ppft, pox, dill, multiprocess, pathos
Successfully installed dill-0.3.6 multiprocess-0.70.14 pathos-0.3.0 pox-0.3.2 ppft-1.7.6.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
dill==0.3.6
multiprocess==0.70.14
pathos==0.3.0
pox==0.3.2
ppft==1.7.6.6
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@An error has occured during the function execution
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/ppft/__main__.py", line 99, in run
__result = __f(*__args)
File "<string>", line 2, in f
ValueError: Error raised!!
Hmm. That's helpful... and so weird. Thanks I'll see if I can reproduce it.
Thanks for making this great library! I have a question regarding to how exception handling could be done with
ParallelPool
. UsingProcessPool
, errors can be caught as normal. However withParallelPool
, the error couldn't be caught likeProcessPool
see below. It seems like the error of all three worker processes are printed out to stderr directly, but none of then are caught.We also noticed that some binary strings are printed out with the error messages. There is a "%" symbol in the end of the example below, but in real life, we got something like this:
I'm wondering if the
ParallelPool
has its own way of error handling and please point me to it if it is already documented!