Unsupported pickle protocol: 5

[stsievert]

I am following Dask's HyperbandSearchCV example verbatim (each cell is copy/pasted into my notebook).

When I get to the line that does the computation, search.fit(X, y, classes=[0, 1, 2, 3], I get this error:

ValueError: unsupported pickle protocol: 5

Full traceback, printed in notebook

``` distributed.protocol.core - CRITICAL - Failed to deserialize Traceback (most recent call last): File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/core.py", line 130, in loads value = _deserialize(head, fs, deserializers=deserializers) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 302, in deserialize return loads(header, frames) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads return pickle.loads(x, buffers=buffers) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads return pickle.loads(x) ValueError: unsupported pickle protocol: 5 distributed.utils - ERROR - unsupported pickle protocol: 5 Traceback (most recent call last): File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/utils.py", line 656, in log_errors yield File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/client.py", line 1221, in _handle_report msgs = await self.scheduler_comm.comm.read() File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 206, in read allow_offload=self.allow_offload, File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/comm/utils.py", line 87, in from_frames res = _from_frames() File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/comm/utils.py", line 66, in _from_frames frames, deserialize=deserialize, deserializers=deserializers File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/core.py", line 130, in loads value = _deserialize(head, fs, deserializers=deserializers) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 302, in deserialize return loads(header, frames) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads return pickle.loads(x, buffers=buffers) File "/home/stsievert/miniconda3/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads return pickle.loads(x) ValueError: unsupported pickle protocol: 5 ```

I don't see any error messages on the Jupyter output stream in the terminal.

This is on Python 3.7.7:

>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=7, micro=7, releaselevel='final', serial=0)

The relevant portion of history:

(base) [stsievert@submit3 ~]$ history
    2  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    3  bash Miniconda3-latest-Linux-x86_64.sh
    5  source ~/.bashrc
    7  conda install numpy scipy pandas scikit-learn
   12  dask-chtc jupyter run lab
   13  which conda
   14  conda install numpy scipy pandas scikit-learn
   15  conda install dask dask dask-ml distributed -c conda-forge
   17  conda install pytorch torchvision cpuonly -c pytorch
   18  pip install --upgrade git+https://github.com/JoshKarpel/dask-chtc.git

[JoshKarpel]

I've been hitting this myself while testing. It's a Python 3.7 vs. 3.8 compatibility problem in pickle. 3.8 adds protocol version 5, and which 3.7 does not support. If I remember correctly, daskdev/dask is on 3.8 now, so you'll need to upgrade your client-side Python to 3.8 as well.

Related Dask issue, where it looks like they concluded that there wasn't an easy fix on their side and you just need to make sure not to cross the protocol boundary: https://github.com/dask/dask/issues/6007

On our end, since we're sitting right at this boundary, this is probably worth adding a note in the docs. Maybe a good candidate for the first entry in a "troubleshooting" page?

[stsievert]

Maybe a good candidate for the first entry in a "troubleshooting" page?

I'll make Python 3.8 a requirement in #28. I've found it easiest to install a new version of Python in a conda env (like in #25).

[stsievert]

Nevermind, I'll leave the Python 3.8 requirement out; it would require documenting the creation of a virtual env; however, #25 already does that.

[JoshKarpel]

"Resolved" by https://github.com/JoshKarpel/dask-chtc/commit/b01d7feae74b68f9add601786df12e1b0b13b33d, but I want to keep this in mind for #25 . We should make it easy to keep your Python version in sync.

[stsievert]

Why not raise an error if Python <3.8 is installed? Or use Dask's ability to version check?

[JoshKarpel]

We can't guarantee that users can use Python >=3.8 for the rest of their code (unlikely, but possible). Dask itself seems unwilling to add a hard incompatibility check for this, so I'd prefer to follow their lead and let the error happen.

One thing I have been a little confused about is that sometimes I get the version mismatch text from Dask when starting workers, but sometimes I don't. Haven't looked into that.