uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

multiprocessing.pool not compatible with -OO flag and PyInstaller #159

Closed b2m closed 5 years ago

b2m commented 5 years ago

Environment

Pathos Version: 0.2.2.1 Python Version: 3.6.6

Problem description

I am trying to generate an optimized python executable using PyInstaller. As with the optimizing the docstrings get removed the following line from multiprocessing.py produces an error when starting the executable.

The "offending" code

__init__.__doc__ = AbstractWorkerPool.__init__.__doc__ + __init__.__doc__

The error message

TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'

For reproduction Here is a script to reproduce the error using a docker image.

# on your command line
docker run -it cdrx/pyinstaller-linux:python3 /bin/bash
# in the docker container
pip install pathos
echo "from pathos.multiprocessing import Pool
if __name__ == '__main__':
     with Pool() as pool:
        pool.map(print, range(10))" > testscript.py
python3 -OO -m PyInstaller  --onefile --noconfirm --clean --name test-pathos testscript.py
./dist/test-pathos

Workaround

Use only -O for optimization with the consequence of having all the docstrings in your executable.

mmckerns commented 5 years ago

I think I've addressed your issue in abfebcd8f282ad1d8e38d2fa0623bb7819c0c75d. If that doesn't work, please reopen the ticket and comment. Thanks for the detailed bug report.

b2m commented 5 years ago

Just confirming that the reported bug is solved on the current master branch.