Closed ikerforce closed 3 years ago
As you say it looks like the dumps_msgpack
method was removed in dask/distributed#4677 and dask/dask#7525.
It also look like distributed 2021.4.1
should depend on dask 2021.4.1
and that is being discussed here.
As a workaround could you ensure you have the latest versions of both dask
and distributed
installed. It looks like you have an older version of dask
in your environment.
I followed your instruction and installed with conda install -c conda-forge dask=2021.4.1 dask-core=2021.4.1 distributed=2021.4.1 dask-yarn
and the issue is gone.
Hope that the default conda install works soon.
Thanks!
That's great. I'm going to close this out as worked around, and things should be addressed in the conda recipe soon.
The following script fails when running it con EMR and HDInsight.
What happened:
The error is the following:
What you expected to happen:
Correct execution of the code as in my local computer (without dask-yarn).
Anything else we need to know?: I was able to get around this error by checking your change-logs and realising that
dumps_msgpack
was removed in the last distributed version. However I followed the exact steps from the offical latest documentation, so I believe this should be corrected or a note posted advising to use distributed 2021.4.0 instead of the default 2021.4.1.Environment: