prophet on Dask: Exception: No module named 'dill'

[eracle]

Hello, I am having this error using Prophet with dask parallel feature.

[W 2021-04-18 16:45:33,994] Trial 0 failed because of the following error: Exception("No module named 'dill'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/optuna/_optimize.py", line 217, in _run_trial
    value_or_values = func(trial)
  File "<ipython-input-16-968766504bad>", line 46, in objective
    horizon = f'{days_in_the_future} days', parallel="dask")
  File "/usr/local/lib/python3.7/dist-packages/fbprophet/diagnostics.py", line 168, in cross_validation
    df, model = pool.scatter([df, model])
  File "/usr/local/lib/python3.7/dist-packages/distributed/client.py", line 2187, in scatter
    hash=hash,
  File "/usr/local/lib/python3.7/dist-packages/distributed/client.py", line 844, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/usr/local/lib/python3.7/dist-packages/distributed/utils.py", line 353, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.7/dist-packages/distributed/utils.py", line 336, in f
    result[0] = yield future
  File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/usr/local/lib/python3.7/dist-packages/distributed/client.py", line 2074, in _scatter
    timeout=timeout,
  File "/usr/local/lib/python3.7/dist-packages/distributed/core.py", line 862, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/distributed/core.py", line 661, in send_recv
    raise exc.with_traceback(tb)
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/core.py", line 501, in handle_comm
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/scheduler.py", line 5024, in scatter
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils_comm.py", line 150, in scatter_to_workers
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils.py", line 237, in All
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/core.py", line 862, in send_recv_from_rpc
  File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/core.py", line 663, in send_recv
Exception: No module named 'dill'

Here the output from client.get_versions(check=True):

{
   "scheduler":{
      "host":{
         "python":"3.7.10.final.0",
         "python-bits":64,
         "OS":"Linux",
         "OS-release":"4.14.225-168.357.amzn2.x86_64",
         "machine":"x86_64",
         "processor":"",
         "byteorder":"little",
         "LC_ALL":"C.UTF-8",
         "LANG":"C.UTF-8"
      },
      "packages":{
         "python":"3.7.10.final.0",
         "dask":"2021.04.0",
         "distributed":"2021.04.0",
         "msgpack":"1.0.2",
         "cloudpickle":"1.6.0",
         "tornado":"6.1",
         "toolz":"0.11.1",
         "numpy":"1.20.2",
         "lz4":"None",
         "blosc":"None"
      }
   },
   "workers":{

   },
   "client":{
      "host":{
         "python":"3.7.10.final.0",
         "python-bits":64,
         "OS":"Linux",
         "OS-release":"4.19.112+",
         "machine":"x86_64",
         "processor":"x86_64",
         "byteorder":"little",
         "LC_ALL":"None",
         "LANG":"en_US.UTF-8"
      },
      "packages":{
         "python":"3.7.10.final.0",
         "dask":"2021.04.0",
         "distributed":"2021.04.0",
         "msgpack":"1.0.2",
         "cloudpickle":"1.6.0",
         "tornado":"6.1",
         "toolz":"0.11.1",
         "numpy":"1.19.5",
         "lz4":"None",
         "blosc":"None"
      }
   }
}

The problem already happened using Airflow and Dask here I am using fbprophet 0.7.2

[eracle]

I solved installing dill on dask workers.

[rpanai]

@facebook Can you close it?