coiled login fails on my laptop

murphyk commented 3 years ago

I am using a macbook pro (catalina, 10.15.7 ). I typed this

pip install coiled --upgrade
coiled login --token <token>

and get this error

TypeError: __init__() got an unexpected keyword argument 'show_default'

Full output

pip install coiled --upgrade
Collecting coiled
  Downloading coiled-0.0.36-py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 3.3 MB/s 
Requirement already satisfied: dask[complete]>=2.23.0 in /Users/kpmurphy/anaconda3/lib/python3.7/site-packages (from coiled) (2020.12.0)
Collecting boto3
...
Successfully installed boto3-1.16.52 cloudpickle-1.6.0 coiled-0.0.36 commonmark-0.9.1 distributed-2021.1.1 pandas-1.2.1 rich-9.10.0 s3transfer-0.3.4

(base) kpmurphy-macbookpro% 
(base) kpmurphy-macbookpro% coiled login --token <token>
Traceback (most recent call last):
  File "/Users/kpmurphy/anaconda3/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/Users/kpmurphy/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/kpmurphy/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/kpmurphy/anaconda3/lib/python3.7/site-packages/click/core.py", line 1135, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/Users/kpmurphy/anaconda3/lib/python3.7/site-packages/click/core.py", line 639, in make_context
    ctx = Context(self, info_name=info_name, parent=parent, **extra)
TypeError: __init__() got an unexpected keyword argument 'show_default'

FabioRosado commented 3 years ago

Hello @murphyk thank you for raising this issue, and I'm sorry you are facing some problems. Please allow me some time to test things to see if I can understand what is happening. When you were using coiled 0.0.35, it worked fine, right?

I will give you an update on this issue. Could I ask you to log off from the coiled website and log back in? This will refresh your access token and make your old one unusable.

As a work around, you can edit the file located on ~/.config/dask/coiled.yaml and add the following (this file probably already contains something if you used coiled before).

coiled:
  account: null
  backend-options: null
  server: https://cloud.coiled.io
  token: <your token>
  user: <your user name>

FabioRosado commented 3 years ago

I had some problems reproducing this issue mostly because I am on BigSur and Python 3.8 already. I did create a conda environment with python 3.7 and tested the login command, but that worked without any problems.

Could I ask you to create a new conda environment, install coiled and see if that fixes the problem? 🤔

jrbourbeau commented 3 years ago

@murphyk could you also let us know what version of click you're using?

murphyk commented 3 years ago

@jrbourbeau : For click, I seem to be using v7.0:

 more /Users/kpmurphy/anaconda3/lib/python3.7/site-packages/click/__init__.py
...
__version__ = '7.0'

@FabioRosado : "When you were using coiled 0.0.35, it worked fine, right?". I have never (knowingly) used coiled before (I had never heard of it...).

murphyk commented 3 years ago

I made a new conda environment and was able to install coiled and get a jupyter notebook to talk to a client in the cloud :). However I get a (file reading?) error when running the NYC taxi example at https://cloud.coiled.io/murphyk/clusters:

---------------------------------------------------------------------------
KilledWorker                              Traceback (most recent call last)
<ipython-input-2-9c89b60c6aab> in <module>
     13 ).persist()
     14 
---> 15 df.groupby("passenger_count").tip_amount.mean().compute()

~/anaconda3/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs)
    277         dask.base.compute
    278         """
--> 279         (result,) = compute(self, traverse=False, **kwargs)
    280         return result
    281 

~/anaconda3/lib/python3.7/site-packages/dask/base.py in compute(*args, **kwargs)
    565         postcomputes.append(x.__dask_postcompute__())
    566 
--> 567     results = schedule(dsk, keys, **kwargs)
    568     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    569 

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2682                     should_rejoin = False
   2683             try:
-> 2684                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2685             finally:
   2686                 for f in futures.values():

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1997                 direct=direct,
   1998                 local_worker=local_worker,
-> 1999                 asynchronous=asynchronous,
   2000             )
   2001 

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    838         else:
    839             return sync(
--> 840                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    841             )
    842 

~/anaconda3/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    338     if error[0]:
    339         typ, exc, tb = error[0]
--> 340         raise exc.with_traceback(tb)
    341     else:
    342         return result[0]

~/anaconda3/lib/python3.7/site-packages/distributed/utils.py in f()
    322             if callback_timeout is not None:
    323                 future = asyncio.wait_for(future, callback_timeout)
--> 324             result[0] = yield future
    325         except Exception as exc:
    326             error[0] = sys.exc_info()

~/.local/lib/python3.7/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1856                             exc = CancelledError(key)
   1857                         else:
-> 1858                             raise exception.with_traceback(traceback)
   1859                         raise exc
   1860                     if errors == "skip":

KilledWorker: ("('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 469)", <Worker 'tls://10.2.13.215:37789', name: murphyk-5218-worker-4-6445f5, memory: 0, processing: 154>)

jrbourbeau commented 3 years ago

Thanks for the click version info, that helps. I'm able to reproduce the issue using the latest coiled release (v0.0.36) and click==7.0. Upgrade click to the latest release (v7.1.2) resolves the issue for me locally. @murphyk could you upgrade click and give coiled login another try?

FabioRosado commented 3 years ago

Regarding the KilledWorker issue, could you go to the Cluster logs and check what happened? Feel free to post them here if you need our help.

murphyk commented 3 years ago

For killed worker, this is what jupyter looks like Screen Shot 2021-02-08 at 2 08 24 PM and this is what the dashboard looks like Screen Shot 2021-02-08 at 2 07 59 PM Clicking on the top left green logs tab gives this:

distributed.scheduler - INFO - Clear task state

distributed.scheduler - INFO - Scheduler at: tls://10.2.12.241:8786

distributed.scheduler - INFO - dashboard at: :8787

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.96:36171', name: murphyk-5225-worker-1-9c973b, memory: 0, processing: 59>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.96:36171

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.107:33713', name: murphyk-5225-worker-3-2af2c0, memory: 0, processing: 53>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.107:33713

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.114:36913', name: murphyk-5225-worker-2-18e347, memory: 0, processing: 155>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.114:36913

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.185:43547', name: murphyk-5225-worker-7-f8c3bd, memory: 0, processing: 48>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.185:43547

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.11:36725', name: murphyk-5225-worker-4-0f2520, memory: 0, processing: 79>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.11:36725

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.220:43241', name: murphyk-5225-worker-5-83a3af, memory: 0, processing: 68>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.220:43241

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.34:42547', name: murphyk-5225-worker-6-28d03b, memory: 0, processing: 427>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.34:42547

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.229:33575', name: murphyk-5225-worker-9-5ec1e3, memory: 0, processing: 227>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.229:33575

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.105:43991', name: murphyk-5225-worker-10-01402c, memory: 0, processing: 94>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.105:43991

distributed.scheduler - INFO - Receive client connection: Client-36af9a9c-6a59-11eb-a9e8-88e9fe624367

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.44:40711', name: murphyk-5225-worker-8-84296e, memory: 0, processing: 117>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.44:40711

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.185:43547', name: murphyk-5225-worker-7-f8c3bd, memory: 0, processing: 48>

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.12.107:33713', name: murphyk-5225-worker-3-2af2c0, memory: 0, processing: 53>

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.96:36171', name: murphyk-5225-worker-1-9c973b, memory: 0, processing: 59>

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.220:43241', name: murphyk-5225-worker-5-83a3af, memory: 0, processing: 68>

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.12.11:36725', name: murphyk-5225-worker-4-0f2520, memory: 0, processing: 79>

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.13.105:43991', name: murphyk-5225-worker-10-01402c, memory: 0, processing: 94>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 27) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 3) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 290) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.12.44:40711', name: murphyk-5225-worker-8-84296e, memory: 0, processing: 117>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 279) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 350) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 29) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 172) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 434) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.13.114:36913', name: murphyk-5225-worker-2-18e347, memory: 0, processing: 155>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 218) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 424) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 333) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 152) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 423) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 273) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 353) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 371) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 431) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.13.229:33575', name: murphyk-5225-worker-9-5ec1e3, memory: 0, processing: 227>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 0) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 332) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 440) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 335) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 207) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 226) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 64) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 362) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 404) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 416) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 343) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 432) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 108) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 327) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 455) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 239) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 233) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 119) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 145) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 261) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 318) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 74) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 352) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 117) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 99) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 235) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 83) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.13.34:42547', name: murphyk-5225-worker-6-28d03b, memory: 0, processing: 427>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 425) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 443) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 9) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 260) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 53) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 334) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 453) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 251) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 28) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 36) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 109) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 18) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 161) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 20) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 154) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 270) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 462) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 192) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 347) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 130) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 407) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 398) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 54) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 75) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 138) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 413) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 137) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 388) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 92) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 341) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 468) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 305) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 143) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 470) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 326) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 182) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 451) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 325) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 2) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 414) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 306) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 199) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 66) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 4) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 309) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 219) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 129) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 59) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 41) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 338) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 266) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 194) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 121) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 58) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 427) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 373) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 319) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 265) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 210) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 157) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 102) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 62) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 242) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 65) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 272) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 406) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 217) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 63) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 324) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 162) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 84) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 372) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 237) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 101) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 68) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 383) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 275) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 167) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 314) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 56) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 191) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 460) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 208) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 315) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 1) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 363) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 183) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 71) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 209) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 415) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 10) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 441) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 171) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 116) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 155) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 299) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 107) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 422) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.220:43241', name: murphyk-5225-worker-5-83a3af, memory: 0, processing: 329>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.220:43241

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.220:43241', name: murphyk-5225-worker-5-83a3af, memory: 0, processing: 329>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 85) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 293) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 297) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 186) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 243) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 387) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 12) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 220) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 174) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 282) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 203) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 131) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 221) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 146) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 252) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 55) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 384) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 90) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 292) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 13) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 378) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 311) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 229) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 76) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 136) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 444) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 253) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 245) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 120) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 317) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 396) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 379) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 46) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 238) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 11) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 128) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 31) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 190) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 127) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 112) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 454) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 144) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 418) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 201) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 163) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 365) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 419) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 401) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 148) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 21) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 158) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 337) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 185) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 24) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 228) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 225) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 417) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 276) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 215) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 87) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 39) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 38) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 283) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 189) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 57) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 258) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 264) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 361) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 15) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 86) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 280) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 390) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 354) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 391) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 49) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 456) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 224) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 408) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 175) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 19) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 445) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 405) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 69) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 33) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 364) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 263) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 82) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 351) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 216) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 446) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 329) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 37) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 95) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 459) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 147) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 72) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 346) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 301) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 227) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 435) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 348) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 50) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 366) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 442) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 40) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 256) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 80) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 100) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 437) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 461) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 438) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 198) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 360) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 271) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 202) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 399) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 42) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 113) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 400) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 168) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 30) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 47) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 389) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 91) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 135) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 170) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 310) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 323) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 316) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 469) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 255) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 5) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 257) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 344) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 153) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 345) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 294) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 298) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 281) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 149) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 134) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 165) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 173) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 262) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 254) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 374) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 184) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 23) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.185:43547', name: murphyk-5225-worker-7-f8c3bd, memory: 0, processing: 181>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.185:43547

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.107:33713', name: murphyk-5225-worker-3-2af2c0, memory: 0, processing: 16>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.107:33713

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.96:36171', name: murphyk-5225-worker-1-9c973b, memory: 0, processing: 16>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.96:36171

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.185:43547', name: murphyk-5225-worker-7-f8c3bd, memory: 0, processing: 181>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 308) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 231) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 448) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 140) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 349) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 328) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 307) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 291) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 206) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 330) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 369) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 240) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 393) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 125) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 370) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 284) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 320) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 397) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 464) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 376) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 244) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 375) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 200) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 193) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 51) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 367) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 123) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 103) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 212) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 339) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 195) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 73) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 392) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 286) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 426) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 420) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 34) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 166) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 289) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 150) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 357) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 67) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 181) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 45) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 356) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 342) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 321) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 222) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 447) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 96) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 402) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 267) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 88) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 114) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 457) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 248) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 94) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 196) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 176) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 285) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 139) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 81) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 111) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 97) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 288) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 78) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 178) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 428) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 16) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 385) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 118) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 300) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 104) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 249) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 410) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 380) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 355) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 450) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 439) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 381) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 452) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 180) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 358) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 465) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 105) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 411) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 25) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 302) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 429) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 6) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 7) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 295) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 14) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 466) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 213) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 433) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 246) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 436) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 463) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 303) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 177) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 234) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 236) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 274) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 77) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 247) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 382) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 204) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 394) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 187) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 211) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 409) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 110) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 277) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 48) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 169) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 32) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 141) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 93) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 159) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 164) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 60) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 336) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 268) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 312) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 43) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 259) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 122) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 79) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 132) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 230) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 126) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 156) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 22) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.105:43991', name: murphyk-5225-worker-10-01402c, memory: 0, processing: 15>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.105:43991

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.34:42547', name: murphyk-5225-worker-6-28d03b, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.34:42547

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.229:33575', name: murphyk-5225-worker-9-5ec1e3, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.229:33575

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.114:36913', name: murphyk-5225-worker-2-18e347, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.114:36913

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.44:40711', name: murphyk-5225-worker-8-84296e, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.44:40711

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.11:36725', name: murphyk-5225-worker-4-0f2520, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.11:36725

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.11.96:36171', name: murphyk-5225-worker-1-9c973b, memory: 0, processing: 16>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 250) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 35) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 232) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 115) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 205) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 188) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 278) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 133) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 61) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 52) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 241) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 386) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 359) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 197) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 287) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 304) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.12.107:33713', name: murphyk-5225-worker-3-2af2c0, memory: 0, processing: 16>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 322) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 8) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 449) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 26) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 395) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 296) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 377) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 214) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 331) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 151) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 403) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 160) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 44) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 421) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 313) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 223) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Remove worker <Worker 'tls://10.2.13.105:43991', name: murphyk-5225-worker-10-01402c, memory: 0, processing: 15>

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 17) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 179) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 368) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 467) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 430) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 70) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 340) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 412) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 106) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 89) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 124) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 458) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 142) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 98) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Task ('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 269) marked as failed because 3 workers died while trying to run it

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.220:43241', name: murphyk-5225-worker-5-83a3af, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.220:43241

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.185:43547', name: murphyk-5225-worker-7-f8c3bd, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.185:43547

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.96:36171', name: murphyk-5225-worker-1-9c973b, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.96:36171

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.107:33713', name: murphyk-5225-worker-3-2af2c0, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.107:33713

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.13.105:43991', name: murphyk-5225-worker-10-01402c, memory: 0, processing: 0>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.13.105:43991

FabioRosado commented 3 years ago

Thank you for the update, could you add the logs of a worker as well? That should tell us what happened.

I know that you are getting started with the quick start example. I want to point that it's always a good idea to have the same dependencies locally and on the cloud. You can do two things here, use coiled install coiled/default-py37 to create a new software environment locally with all the dependencies used by coiled. Or you could add them manually.

I'm wondering if the version mismatch could be the cause of your workers dying 🤔

Let me know if you would like to chat about this 1-1 I'm happy to help you get started with Coiled

murphyk commented 3 years ago

I updated click and did coiled install coiled/default-py37 to get a new environment. I restarted the notebook and ran it. Briefly the dashboard showed some activity and I was excited. Shortyl thereafter(maybe after reading first csv file?) it failed as before with a killed worker error. Details below (with logs). Sorry to say, but I'm giving up (i was just curious about coiled, did not plan to pay for it...).

KilledWorker                              Traceback (most recent call last)
<ipython-input-2-9c89b60c6aab> in <module>
     13 ).persist()
     14 
---> 15 df.groupby("passenger_count").tip_amount.mean().compute()

~/anaconda3/lib/python3.7/site-packages/dask/base.py in compute(self, **kwargs)
    277         dask.base.compute
    278         """
--> 279         (result,) = compute(self, traverse=False, **kwargs)
    280         return result
    281 

~/anaconda3/lib/python3.7/site-packages/dask/base.py in compute(*args, **kwargs)
    565         postcomputes.append(x.__dask_postcompute__())
    566 
--> 567     results = schedule(dsk, keys, **kwargs)
    568     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    569 

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2682                     should_rejoin = False
   2683             try:
-> 2684                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2685             finally:
   2686                 for f in futures.values():

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1997                 direct=direct,
   1998                 local_worker=local_worker,
-> 1999                 asynchronous=asynchronous,
   2000             )
   2001 

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    838         else:
    839             return sync(
--> 840                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    841             )
    842 

~/anaconda3/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    338     if error[0]:
    339         typ, exc, tb = error[0]
--> 340         raise exc.with_traceback(tb)
    341     else:
    342         return result[0]

~/anaconda3/lib/python3.7/site-packages/distributed/utils.py in f()
    322             if callback_timeout is not None:
    323                 future = asyncio.wait_for(future, callback_timeout)
--> 324             result[0] = yield future
    325         except Exception as exc:
    326             error[0] = sys.exc_info()

~/.local/lib/python3.7/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

~/anaconda3/lib/python3.7/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1856                             exc = CancelledError(key)
   1857                         else:
-> 1858                             raise exception.with_traceback(traceback)
   1859                         raise exc
   1860                     if errors == "skip":

KilledWorker: ("('read-csv-4944a4e90a27ee8f4883e6cac95c831a', 469)", <Worker 'tls://10.2.13.201:41623', name: murphyk-5232-worker-8-885ec6, memory: 0, processing: 427>)

```

Main logs

istributed.scheduler - INFO - Clear task state

distributed.scheduler - INFO - Scheduler at: tls://10.2.13.135:8786

distributed.scheduler - INFO - dashboard at: :8787

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.92:43859', name: murphyk-5233-worker-10-c28cde, memory: 0, processing: 427>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.92:43859

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.11.161:43023', name: murphyk-5233-worker-8-867893, memory: 0, processing: 53>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.11.161:43023

distributed.scheduler - INFO - Register worker <Worker 'tls://10.2.12.27:33925', name: murphyk-5233-worker-7-7ef642, memory: 0, processing: 95>

distributed.scheduler - INFO - Starting worker compute stream, tls://10.2.12.27:33925 ...


Worker 1 logs

istributed.worker - INFO - Start worker at: tls://10.2.11.161:43023

distributed.worker - INFO - Listening to: tls://10.2.11.161:43023

distributed.worker - INFO - dashboard at: 10.2.11.161:42099

distributed.worker - INFO - Waiting to connect to: tls://ip-10-2-13-135.us-east-2.compute.internal:8786

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Threads: 4

distributed.worker - INFO - Memory: 17.18 GB

distributed.worker - INFO - Local Directory: /dask-worker-space/dask-worker-space/worker-wxnjki6h

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Waiting to connect to: tls://ip-10-2-13-135.us-east-2.compute.internal:8786

distributed.worker - INFO - Registered to: tls://ip-10-2-13-135.us-east-2.compute.internal:8786

distributed.worker - INFO - -------------------------------------------------

distributed.worker - ERROR - Can't get attribute 'PackedFunctionCall' on <module 'dask.blockwise' from '/opt/conda/envs/coiled/lib/python3.7/site-packages/dask/blockwise.py'> Traceback (most recent call last): File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/worker.py", line 991, in handle_scheduler comm, every_cycle=[self.ensure_communicating, self.ensure_computing] File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/core.py", line 554, in handle_stream msgs = await comm.read() File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/comm/tcp.py", line 216, in read allow_offload=self.allow_offload, File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/comm/utils.py", line 80, in from_frames res = _from_frames() File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/comm/utils.py", line 64, in _from_frames frames, deserialize=deserialize, deserializers=deserializers File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/core.py", line 151, in loads value = _deserialize(head, fs, deserializers=deserializers) File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 374, in deserialize return loads(header, frames) File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 70, in pickle_loads return pickle.loads(x, buffers=buffers) File "/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads return pickle.loads(x) AttributeError: Can't get attribute 'PackedFunctionCall' on <module 'dask.blockwise' from '/opt/conda/envs/coiled/lib/python3.7/site-packages/dask/blockwise.py'>

distributed.worker - INFO - Connection to scheduler broken. Reconnecting...

distributed.worker - INFO - -------------------------------------------------

distributed.worker - WARNING - Mismatched versions found +-------------+---------------+---------------+------------------------------------+ | Package | This Worker | scheduler | workers | +-------------+---------------+---------------+------------------------------------+ | blosc | 1.10.2 | 1.10.2 | {None, '1.10.2'} | | dask | 2021.02.0 | 2021.02.0 | {'2020.12.0', '2021.02.0'} | | distributed | 2021.02.0 | 2021.02.0 | {'2021.01.1', '2021.02.0'} | | lz4 | 3.1.3 | 3.1.3 | {'3.1.3', None} | | numpy | 1.20.0 | 1.20.0 | {'1.20.0', '1.18.1'} | | python | 3.7.9.final.0 | 3.7.9.final.0 | {'3.7.9.final.0', '3.7.6.final.0'} | | toolz | 0.11.1 | 0.11.1 | {'0.10.0', '0.11.1'} | | tornado | 6.1 | 6.1 | {'6.1', '6.0.4'} | +-------------+---------------+---------------+------------------------------------+