search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.57k stars 716 forks source link

Can't figure out what Dask wants in terms of msgpack/msgpack-python #3876

Closed rjurney closed 4 years ago

rjurney commented 4 years ago

I can't figure out what Dask wants in terms of msgpack/msgpack-python and I've read all about it. I am trying to run it in distributed mode, but my workers die. I fixed this once but lost the solution and the default setup doesn't work so I have to run workers manually.

The command I run:

nohup dask-scheduler &
dask-worker --nprocs 2 --nthreads 2 127.0.0.1:8786

What happened: I get this result:

distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:38591
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:38591
distributed.worker - INFO -          dashboard at:            127.0.0.1:38859
distributed.worker - INFO - Waiting to connect to:       tcp://127.0.0.1:8786
distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:40635
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:40635
distributed.worker - INFO -          dashboard at:            127.0.0.1:45265
distributed.worker - INFO - Waiting to connect to:       tcp://127.0.0.1:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                    8.36 GB
distributed.worker - INFO -       Local Directory: /data/matcha.doodle.backend/dask-worker-space/worker-0p7c8zw4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                    8.36 GB
distributed.worker - INFO -       Local Directory: /data/matcha.doodle.backend/dask-worker-space/worker-ek57dqzs
distributed.worker - INFO - -------------------------------------------------
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 713, in run
    await worker
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 1060, in start
    await self._register_with_scheduler()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 844, in _register_with_scheduler
    response = await future
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 713, in run
    await worker
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 1060, in start
    await self._register_with_scheduler()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/worker.py", line 844, in _register_with_scheduler
    response = await future
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/tcp.py", line 212, in read
    frames, deserialize=self.deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 75, in from_frames
    res = _from_frames()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 107, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 199, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_opts)
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP  local=tcp://127.0.0.1:34912 remote=tcp://127.0.0.1:8786>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP  local=tcp://127.0.0.1:34914 remote=tcp://127.0.0.1:8786>
distributed.nanny - ERROR - Failed while trying to start worker process: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.nanny - ERROR - Failed while trying to start worker process: unpackb() got an unexpected keyword argument 'strict_map_key'
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/dask-worker", line 11, in <module>
    sys.exit(go())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 436, in go
    main()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 422, in main
    loop.run_sync(run)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/ioloop.py", line 532, in run_sync
    return future_cell[0].result()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 416, in run
    await asyncio.gather(*nannies)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 251, in start
    response = await self.instantiate()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 334, in instantiate
    result = await self.process.start()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 528, in start
    msg = await self._wait_until_connected(uid)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/nanny.py", line 643, in _wait_until_connected
    raise msg
TypeError: exceptions must derive from BaseException

What you expected to happen:

I expect the workers to live. Instead they die. I can't figure out what Dask wants in terms of msgpack.

Anything else we need to know?:

Can someone please tell me how to setup my environment in a way Dask likes regarding msgpack/msgpack-python? I first used pip to install both then found a bug fix that said use conda so I did. That did not help.

Environment:

dask==2.14.0
dask-glm==0.2.0
dask-ml==1.5.0

pip freeze:

alabaster==0.7.12
anaconda-client==1.7.2
anaconda-project==0.8.3
appdirs==1.4.4
argh==0.26.2
asn1crypto==1.3.0
astroid==2.3.3
astropy==4.0.1.post1
atomicwrites==1.3.0
atpublic==1.0
attrs==19.3.0
autopep8==1.4.4
autovizwidget==0.15.0
awscli==1.18.71
Babel==2.8.0
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.8.2
bitarray==1.2.1
bkcharts==0.2
bleach==3.1.4
blessings==1.7
blis==0.2.4
bokeh==2.0.1
boto==2.49.0
boto3==1.13.13
botocore==1.16.21
Bottleneck==1.3.2
bs4==0.0.1
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
click==7.1.1
cloudpickle==1.3.0
clyent==1.2.2
colorama==0.4.3
ConceptNet==5.7.0
ConfigArgParse==1.2.3
configobj==5.0.6
configparser==5.0.0
contextlib2==0.6.0.post1
coverage==5.1
cryptography==2.8
cycler==0.10.0
cymem==2.0.2
Cython==0.29.15
cytoolz==0.10.1
dask==2.14.0
dask-glm==0.2.0
dask-ml==1.5.0
dataclasses==0.6
datadog==0.36.0
datrie==0.8.2
decorator==4.4.2
defusedxml==0.6.0
Deprecated==1.2.10
diff-match-patch==20181111
distributed==2.14.0
distro==1.5.0
dnspython==1.16.0
docker-pycreds==0.4.0
docutils==0.15.2
dpath==2.0.1
dvc==0.94.0
email-validator==1.1.1
entrypoints==0.3
enum34==1.1.10
environment-kernels==1.1.1
et-xmlfile==1.0.1
fastai==1.0.61
fastcache==1.1.0
fastparquet==0.4.0
fastprogress==0.2.2
filelock==3.0.12
flake8==3.7.9
Flask==1.1.1
flatten-json==0.1.7
flufl.lock==3.2
frozendict==1.2
fsspec==0.7.1
ftfy==5.7
funcy==1.14
future==0.18.2
gevent==1.4.0
gitdb==4.0.5
GitPython==3.1.3
glob2==0.7
gmpy2==2.0.8
gpustat==0.6.0
gql==0.2.0
grandalf==0.6
graphql-core==1.1
graphviz==0.14
greenlet==0.4.15
h5py==2.10.0
hdijupyterutils==0.15.0
HeapDict==1.0.1
html5lib==1.0.1
humanize==2.4.0
hypothesis==5.8.3
idna==2.9
imageio==2.8.0
imagesize==1.2.0
importlib-metadata==1.5.0
inflect==3.0.2
intervaltree==3.0.2
ipykernel==5.1.4
ipyparallel==6.2.4
ipython==7.13.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
iso8601==0.1.12
isort==4.3.21
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.15.2
jeepney==0.4.3
Jinja2==2.11.1
jmespath==0.9.4
joblib==0.14.1
JSON-log-formatter==0.3.0
json5==0.9.4
jsonpath-ng==1.5.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.2
jupyter-console==6.1.0
jupyter-core==4.6.3
jupyterlab==1.2.6
jupyterlab-server==1.1.0
keyring==21.1.1
kiwisolver==1.1.0
langcodes==2.0.0
lazy-object-proxy==1.4.3
libarchive-c==2.8
lief==0.9.0
llvmlite==0.31.0
locket==0.2.0
lxml==4.5.0
marisa-trie==0.7.5
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.1.3
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.0.15
mkl-random==1.1.0
mkl-service==2.3.0
mock==4.0.1
more-itertools==8.2.0
mpmath==1.1.0
msgpack==1.0.0
msgpack-python==0.5.6
multipledispatch==0.6.0
murmurhash==1.0.2
nanotime==0.5.2
nb-conda==2.2.1
nb-conda-kernels==2.2.3
nbconvert==5.6.1
nbdime==2.0.0
nbformat==5.0.4
networkx==2.3
nltk==3.4.5
nose==1.3.7
notebook==6.0.3
numba==0.48.0
numexpr==2.7.1
numpy==1.18.1
numpydoc==0.9.2
nvidia-ml-py3==7.352.0
olefile==0.46
onnx==1.5.0
opencv-python==4.2.0.32
openpyxl==3.0.3
ordered-set==4.0.1
packaging==20.3
pandas==1.0.3
pandoc==1.0.2
pandocfilters==1.4.2
parso==0.5.2
partd==1.1.0
path==13.1.0
pathlib2==2.3.5
pathspec==0.8.0
pathtools==0.1.2
patsy==0.5.1
pep8==1.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.0.0
pkginfo==1.5.0.1
plac==0.9.6
plotly==4.7.1
pluggy==0.13.1
ply==3.11
preshed==2.0.1
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==3.0.4
protobuf==3.12.0
protobuf3-to-dict==0.1.5
psutil==5.7.0
psycopg2==2.7.5
psycopg2-binary==2.8.5
ptyprocess==0.6.0
py==1.8.1
py4j==0.10.7
pyarrow==0.17.1
pyasn1==0.4.8
pycodestyle==2.5.0
pycosat==0.6.3
pycparser==2.20
pycrypto==2.6.1
pycurl==7.43.0.5
pydocstyle==4.0.1
pydot==1.4.1
pyflakes==2.1.1
pygal==2.4.0
PyGithub==1.51
Pygments==2.6.1
pygtrie==2.3.2
PyJWT==1.7.1
pykerberos==1.2.1
pylint==2.4.4
pymongo==3.10.1
pyodbc===4.0.0-unsupported
pyOpenSSL==19.1.0
pyparsing==2.4.6
pyrsistent==0.16.0
PySocks==1.7.1
pyspark==2.4.5
pytest==5.4.1
pytest-arraydiff==0.3
pytest-astropy==0.8.0
pytest-astropy-header==0.1.2
pytest-doctestplus==0.5.0
pytest-openfiles==0.4.0
pytest-remotedata==0.3.2
pytextrank==2.0.2
python-dateutil==2.8.1
python-jsonrpc-server==0.3.4
python-language-server==0.31.9
pytz==2019.3
PyWavelets==1.1.1
pyxdg==0.26
PyYAML==5.3.1
pyzmq==18.1.1
QDarkStyle==2.8
QtAwesome==0.7.0
qtconsole==4.7.2
QtPy==1.9.0
ratelimiter==1.2.0.post0
regex==2020.5.14
requests==2.23.0
requests-kerberos==0.12.0
retrying==1.3.3
rfc3986==1.4.0
rope==0.16.0
rsa==3.4.2
Rtree==0.9.3
ruamel-yaml==0.15.87
ruamel.yaml.clib==0.2.0
s3fs==0.4.0
s3transfer==0.3.3
sacremoses==0.0.43
sagemaker==1.58.3
scikit-image==0.16.2
scikit-learn==0.23.1
scipy==1.4.1
seaborn==0.10.0
SecretStorage==3.1.2
semantic-version==2.8.5
Send2Trash==1.5.0
sentencepiece==0.1.91
sentry-sdk==0.14.4
shortuuid==1.0.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.14.0
smart-open==2.0.0
smdebug-rulesconfig==0.1.2
smmap==3.0.4
snakemake==5.19.0
snowballstemmer==2.0.0
sortedcollections==1.1.2
sortedcontainers==2.1.0
soupsieve==2.0
spacy==2.1.8
sparkmagic==0.15.0
Sphinx==2.4.4
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.1
spyder==4.1.2
spyder-kernels==1.9.0
SQLAlchemy==1.3.15
srsly==0.1.0
statsmodels==0.11.0
subprocess32==3.5.4
summa==1.2.0
sympy==1.5.1
tables==3.6.1
tblib==1.6.0
terminado==0.8.3
testpath==0.4.4
texttable==1.6.2
thinc==7.0.8
threadpoolctl==2.1.0
thrift==0.13.0
tokenizers==0.7.0
toolz==0.10.0
toposort==1.5
torch==1.4.0
torchvision==0.5.0
tornado==6.0.4
tqdm==4.46.1
traitlets==4.3.3
transformers==2.11.0
treelib==1.6.1
typed-ast==1.4.1
typing==3.7.4.1
typing-extensions==3.7.4.1
ujson==1.35
unicodecsv==0.14.1
urllib3==1.25.8
voluptuous==0.11.7
wandb==0.9.0
wasabi==0.2.2
watchdog==0.10.2
wcwidth==0.1.9
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wordfreq==2.3.2
wrapt==1.12.1
wurlitzer==2.0.0
xlrd==1.2.0
XlsxWriter==1.2.8
xlwt==1.3.0
xmltodict==0.11.0
yapf==0.28.0
zc.lockfile==2.0
zict==2.0.0
zipp==2.2.0
mrocklin commented 4 years ago

Hi @rjurney !

Oof, that sounds frustrating. Dask requries msgpack-python >= 0.6.0. It looks like you have version 0.5. A quick conda install dask should bring you up to a sensible configuration.

I'm actually quite surprised that conda let you have that version of msgpack-python installed. We currently pin to >=0.6.0 in the conda recipe.

rjurney commented 4 years ago

@mrocklin The latest version of msgpack-python on PyPi is 0.5.6. See: https://pypi.org/project/msgpack-python/#files

I see newer version here though: https://anaconda.org/anaconda/msgpack-python/files

mrocklin commented 4 years ago

If you're downloading from PyPI then you want the msgpack package: https://pypi.org/project/msgpack/ If you're downloading from conda-forge there is a conflict because there are many msgpack packages. The python one is called msgpack-python.

Regardless, if you follow the install instructions at https://docs.dask.org/en/latest/install.html then your package manager (pip or conda) should ensure that you have the right packages necessary to run Dask.

On Tue, Jun 9, 2020 at 5:17 PM Russell Jurney notifications@github.com wrote:

@mrocklin https://github.com/mrocklin The latest version of msgpack-python on PyPi is 0.5.6. See: https://pypi.org/project/msgpack-python/#files

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3876#issuecomment-641646121, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTC2DYSHQIGPQXJ6BG3RV3GJPANCNFSM4NZ3P4VQ .

rjurney commented 4 years ago

@mrocklin The instructions assume a good version gets installed as a dependency of Dask and it does not. They don't speak as to msgpack.

I have msgpack==1.0.0 after running conda install -c anaconda "msgpack-python" and I get this error:

I can't figure out what Dask wants from me, it makes no sense. I did what I'm supposed to do and the right package is installed.

dask-worker --help|grep -i scheduler
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/bin/dask-worker", line 7, in <module>
    from distributed.cli.dask_worker import go
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/__init__.py", line 3, in <module>
    from .actor import Actor, ActorFuture
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/actor.py", line 6, in <module>
    from .client import Future, default_client
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/client.py", line 42, in <module>
    from .batched import BatchedSend
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/batched.py", line 8, in <module>
    from .core import CommClosedError
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/core.py", line 17, in <module>
    from .comm import (
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/__init__.py", line 25, in <module>
    _register_transports()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/__init__.py", line 16, in _register_transports
    from . import inproc
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/comm/inproc.py", line 13, in <module>
    from ..protocol import nested_deserialize
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/__init__.py", line 5, in <module>
    from .core import dumps, loads, maybe_compress, decompress, msgpack
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/core.py", line 8, in <module>
    from .serialize import serialize, deserialize, Serialize, Serialized, extract_serialize
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 14, in <module>
    from .utils import (
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/distributed/protocol/utils.py", line 15, in <module>
    msgpack.loads(msgpack.dumps(""), raw=False, **msgpack_opts)
AttributeError: module 'msgpack' has no attribute 'loads'
rjurney commented 4 years ago

@mrocklin I removed all msgpack and msgpack-python modules and installed msgpack from pip.

pip freeze shows: msgpack==1.0.0 and cloudpickle==1.3.0

Now I get this error:

TypeError: Data serialized with pickle but only able to deserialize data with ['msgpack']
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP  local=tcp://127.0.0.1:35126 remote=tcp://127.0.0.1:8786>
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>
Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError

Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process
    assert exitcode is not None
AssertionError
mrocklin commented 4 years ago

You shouldn't have to explicitly install msgpack. Can you verify that you've done the instructions in https://docs.dask.org/en/latest/install.html . Your package manager will handle this for you.

On Tue, Jun 9, 2020 at 5:33 PM Russell Jurney notifications@github.com wrote:

@mrocklin https://github.com/mrocklin I removed all msgpack and msgpack-python modules and installed msgpack from pip.

Now I get this error:

TypeError: Data serialized with pickle but only able to deserialize data with ['msgpack']distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.comm.tcp - WARNING - Closing dangling stream in distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxydistributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, **self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3876#issuecomment-641650968, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTAC3NRK7TVMJVZKN23RV3IFHANCNFSM4NZ3P4VQ .

TomAugspurger commented 4 years ago

And I'd recommend creating a clean conda or virtual environment to test out the instructions at https://docs.dask.org/en/latest/install.html. Pip and sometimes conda can struggle to install packages into currently broken environments.

On Tue, Jun 9, 2020 at 10:16 PM Matthew Rocklin notifications@github.com wrote:

You shouldn't have to explicitly install msgpack. Can you verify that you've done the instructions in https://docs.dask.org/en/latest/install.html . Your package manager will handle this for you.

On Tue, Jun 9, 2020 at 5:33 PM Russell Jurney notifications@github.com wrote:

@mrocklin https://github.com/mrocklin I removed all msgpack and msgpack-python modules and installed msgpack from pip.

Now I get this error:

TypeError: Data serialized with pickle but only able to deserialize data with ['msgpack']distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:35126 remote=tcp://127.0.0.1:8786>distributed.dashboard.proxy

  • INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxydistributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>distributed.process - INFO - reaping stray process <SpawnProcess(Dask Worker process (from Nanny), started daemon)>Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(*self._args, *self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run self._target(self._args, **self._kwargs) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 234, in _watch_process assert exitcode is not NoneAssertionError

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/dask/distributed/issues/3876#issuecomment-641650968 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AACKZTAC3NRK7TVMJVZKN23RV3IFHANCNFSM4NZ3P4VQ

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3876#issuecomment-641694658, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAOIQEZQNOXUHTEEFZWQTRV33IPANCNFSM4NZ3P4VQ .

rjurney commented 4 years ago

@TomAugspurger I got it to work in a clean environment, thanks.

sravzpublic commented 2 years ago

I had these package preinstalled before Dask and distributed where installed: pip3 freeze | grep msgpack msgpack==1.0.2 msgpack-python==0.5.6

Dask distributed package pulls in msg pack. To ensure scheduler and workers have same pip packages. Un install msg pack and allow distributed install it.

pip uninstall msgpack-python pip uninstall msgpack pip uninstall "dask[complete]" pip uninstall "distributed"

Install dask and distributed only: pip install --upgrade --no-deps --force-reinstall "dask[complete]" pip install "distributed"

Distributed will pull msgpack Collecting msgpack>=0.6.0 Downloading msgpack-1.0.2-cp38-cp38-manylinux1_x86_64.whl (302 kB) |████████████████████████████████| 302 kB 18.9 MB/s

Client connect should work then

client = Client('xxxx:8786') +-------------+-----------+-----------+-----------+ | Package | client | scheduler | workers | +-------------+-----------+-----------+-----------+ | dask | 2021.11.1 | 2021.11.0 | 2021.11.0 | | distributed | 2021.11.1 | 2021.11.0 | 2021.11.0 | | numpy | 1.20.3 | 1.21.4 | 1.21.4 | | pandas | 1.3.3 | 1.3.4 | 1.3.4 | | toolz | 0.11.1 | 0.11.2 | 0.11.2 |