Connecting to a Client in a ProcessPoolExecutor doesn't work

[basnijholt]

For reasons I want to connect to a ipyparallel.Client within subprocesses.

Unfortunately, the Client always reports a timeout error, even though running client = ipyparallel.Client() in my main process is a fraction of a second.

from concurrent.futures import ProcessPoolExecutor
ex = ProcessPoolExecutor(2)

def connect_client():
    import ipyparallel
    client = ipyparallel.Client(timeout=60)
    # do_stuff()

fut = ex.submit(connect_client)

fut.result()

Traceback:

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "<ipython-input-2-b678834d0b21>", line 6, in connect_client
    client = ipyparallel.Client(timeout=60)
  File "/opt/conda/lib/python3.6/site-packages/ipyparallel/client/client.py", line 495, in __init__
    self._connect(sshserver, ssh_kwargs, timeout)
  File "/opt/conda/lib/python3.6/site-packages/ipyparallel/client/client.py", line 615, in _connect
    raise error.TimeoutError("Hub connection request timed out")
ipyparallel.error.TimeoutError: Hub connection request timed out
"""

The above exception was the direct cause of the following exception:

TimeoutError                              Traceback (most recent call last)
<ipython-input-2-b678834d0b21> in <module>()
      9 fut = ex.submit(connect_client)
     10 
---> 11 fut.result()

/opt/conda/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/opt/conda/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

TimeoutError: Hub connection request timed out

[minrk]

What's pip list?

[basnijholt]

•% ➜ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
alembic (0.9.6)
asn1crypto (0.22.0)
attrs (17.4.0)
backports.functools-lru-cache (1.4)
bcrypt (3.1.4)
bleach (2.0.0)
bokeh (0.12.13)
Bottleneck (1.2.1)
certifi (2017.7.27.1)
cffi (1.10.0)
chardet (3.0.4)
click (6.7)
cloudpickle (0.5.2)
colorama (0.3.9)
conda (4.3.29)
cryptography (2.0.3)
cycler (0.10.0)
dask (0.16.1)
decorator (4.1.2)
deepdish (0.3.4)
dijitso (2016.2.0)
dill (0.2.7.1)
distributed (1.20.2)
emcee (2.2.1)
entrypoints (0.2.3)
execnet (1.3.0)
fastcache (1.0.2)
feedparser (5.2.1)
FFC (2016.2.0)
FIAT (2016.2.0)
gitdb2 (2.0.2)
GitPython (2.1.7)
gmpy2 (2.0.8)
graphviz (0.8)
h5netcdf (0.5.0)
h5py (2.7.1)
heapdict (1.0.0)
holoviews (1.9.2)
hpc05 (1.20+1.g965566c)
html5lib (1.0.1)
idna (2.6)
instant (2016.2.0)
ipykernel (4.7.0)
ipyparallel (6.0.2)
ipython (6.2.1)
ipython-genutils (0.2.0)
ipywidgets (7.1.0)
jedi (0.10.2)
Jinja2 (2.10)
jsonschema (2.6.0)
jupyter-client (5.1.0)
jupyter-cms (0.7.0)
jupyter-console (5.2.0)
jupyter-core (4.4.0)
jupyter-dashboards (0.7.0)
jupyter-dashboards-bundlers (0.9.1)
jupyterhub (0.8.1)
jupyterlab (0.31.0)
jupyterlab-launcher (0.10.2)
kwant (1.3.2)
line-profiler (2.1.2)
locket (0.2.0)
Mako (1.0.7)
MarkupSafe (1.0)
matplotlib (2.1.1)
mistune (0.8.3)
msgpack-python (0.4.8)
nb-conda (2.2.1)
nb-conda-kernels (2.1.0)
nbconvert (5.3.1)
nbdime (0.4.0)
nbformat (4.4.0)
nbserverproxy (0.4)
nbstripout (0.3.1)
netCDF4 (1.3.1)
notebook (5.2.2)
numexpr (2.6.4)
numpy (1.13.3)
olefile (0.44)
pamela (0.3.0)
pandas (0.22.0)
pandocfilters (1.4.1)
param (1.5.1)
paramiko (2.3.1)
partd (0.3.8)
petsc4py (3.7.0)
pexpect (4.3.0)
pickleshare (0.7.4)
Pillow (4.3.0)
pip (9.0.1)
plotly (2.2.3)
pluggy (0.6.0)
ply (3.10)
prompt-toolkit (1.0.15)
psutil (5.4.0)
ptyprocess (0.5.2)
py (1.5.2)
pyasn1 (0.4.2)
pycosat (0.6.2)
pycparser (2.18)
pyflakes (1.6.0)
Pygments (2.2.0)
PyNaCl (1.1.2)
PyOpenGL (3.1.0)
pyOpenSSL (17.2.0)
pyparsing (2.2.0)
PySocks (1.6.7)
pytest (3.3.2)
pytest-cache (1.0)
pytest-flakes (1.0.1)
python-dateutil (2.6.1)
python-editor (1.0.3)
python-gitlab (1.2.0)
python-oauth2 (1.0.1)
pytz (2017.3)
PyYAML (3.12)
pyzmq (16.0.2)
qtconsole (4.3.1)
qtplot (0.1.1.dev9)
QtPy (1.3.1)
requests (2.18.4)
ruamel-yaml (0.11.14)
scipy (1.0.0)
setuptools (36.6.0)
simplegeneric (0.8.1)
six (1.11.0)
slepc4py (3.7.0)
smmap2 (2.0.3)
sortedcontainers (1.5.7)
SQLAlchemy (1.1.13)
sshtunnel (0.1.0)
sympy (1.1.1)
tables (3.4.2)
tblib (1.3.2)
terminado (0.8.1)
testpath (0.3.1)
tinyarray (1.2.0)
toolz (0.8.2)
tornado (4.5.2)
traitlets (4.3.2)
UFL (2016.2.0)
urllib3 (1.22)
vispy (0.5.2)
wcwidth (0.1.7)
webencodings (0.5)
wheel (0.30.0)
Whoosh (2.7.4)
widgetsnbextension (3.1.0)
xarray (0.10.0)
xlrd (1.1.0)
zict (0.1.3)

[minrk]

Is this in an IPython session? If so, I bet it's inheriting the zmq.Context.instance from the parent process, which isn't usable after the fork. I'm a bit surprised that it isn't raising an error, but passing a new context should work:

client = ipp.Client(context=zmq.Context())

[basnijholt]

Yes, it is an IPython session. Your solution worked, thanks!

[minrk]

Thanks! There might be a way to detect the fork and cause zmq.Context.instance() to work after fork (creating a new context), but that would be in pyzmq.