Worker failed to start

[brahma0545]

import distributed print(distributed.__version__) 1.21.2

import tornado print(tornado.version) 4.5.3

from dask.distributed import Client, LocalCluster client = Client()

tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\brahm\Anaconda3\lib\site-packages\tornado\gen.py", line 1069, in run yielded = self.gen.send(value) File "C:\Users\brahm\Anaconda3\lib\site-packages\distributed\deploy\local.py", line 196, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.gen.TimeoutError: Worker failed to start

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\brahm\Anaconda3\lib\site-packages\tornado\gen.py", line 828, in callback result_list.append(f.result()) File "C:\Users\brahm\Anaconda3\lib\site-packages\tornado\concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "C:\Users\brahm\Anaconda3\lib\site-packages\tornado\gen.py", line 1069, in run yielded = self.gen.send(value) File "C:\Users\brahm\Anaconda3\lib\site-packages\distributed\deploy\local.py", line 196, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.gen.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\brahm\Anaconda3\lib\site-packages\tornado\gen.py", line 1069, in run yielded = self.gen.send(value) File "C:\Users\brahm\Anaconda3\lib\site-packages\distributed\deploy\local.py", line 196, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.gen.TimeoutError: Worker failed to start

During handling of the above exception, another exception occurred: ...

[mrocklin]

Can I ask you to try the following?

1. Avoid processes with client = Client(processes=False)
2. See if you can create things locally with the command line? http://dask.pydata.org/en/latest/setup/cli.html

[brahma0545]

Yes, it is working fine :) thank you

[mrocklin]

Do both of those work fine? There is still a problem here, I'd like to track it down.

[brahma0545]

its working on both terminal and jupyter notebook

[mrocklin]

Ah, I apologize, I wasn't clear. These are two different solutions. Either use the CLI (dask-scheduler) or use Client(proceses=False).

Currently you're effectively using Client(processes=False), which seems like it works, which is good. Hopefully this can be effective for you short-term.

If you also have time I would appreciate knowing if the following solution also works.

on the command line

$ dask-scheduler
$ dask-worker localhost:8786  # this will have to be in a different terminal

In Python

from dask.distributed import Client
client = Client('localhost:8786')

[mrocklin]

Any update on this @krishna0545 ?

[j-bennet]

I also have a lot of these with latest pip-installed dask[complete].

In most cases, they are preceded by Found stale lock file and directory:

distributed.diskutils - WARNING - Found stale lock file and directory '/Users/irina/src/talks/2018/daskvsspark/dask-worker-space/worker-BzGzLF', purging
tornado.application - ERROR - Exception in Future <tornado.concurrent.Future object at 0x10c3265d0> after timeout
Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/nanny.py", line 375, in start
    self.process.terminate())
AttributeError: 'NoneType' object has no attribute 'terminate'
distributed.utils - ERROR - Worker failed to start
Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/client.py", line 751, in _start
    yield self.cluster
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1280, in _wrap_awaitable
    _y = _m(*_x)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 210, in __await__
    result = yield self
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1263, in _wrap_awaitable
    _s = yield _y
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/deploy/local.py", line 163, in _start
    yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/deploy/local.py", line 189, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
TimeoutError: Worker failed to start
Failed in 0:01:00.636737.
Traceback (most recent call last):
  File "aggregate_dask.py", line 156, in <module>
    client = Client(processes=not args.no_procs)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/client.py", line 578, in __init__
    self.start(timeout=timeout)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/client.py", line 694, in start
    sync(self.loop, self._start, **kwargs)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/utils.py", line 254, in sync
    six.reraise(*error[0])
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/client.py", line 751, in _start
    yield self.cluster
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1280, in _wrap_awaitable
    _y = _m(*_x)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 210, in __await__
    result = yield self
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1263, in _wrap_awaitable
    _s = yield _y
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/deploy/local.py", line 163, in _start
    yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/concurrent.py", line 260, in result
    raise_exc_info(self._exc_info)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/Users/irina/.pyenv/versions/talks/lib/python2.7/site-packages/distributed/deploy/local.py", line 189, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start

dask and tornado versions:

dask==0.17.2
distributed==1.21.5
tornado==5.0.1

The solution with processes=False works, but I do want to use processes. The solution with starting workers by hand and using Client('localhost:8786') works.

[mrocklin]

Just verifying what you're saying. Creating a dask-scheduler and dask-workers manually works fine, but creating those by calling Client() with no arguments doesn't work? Does it always fail to work, or does it only fail to work sometimes? From your paths it looks like you are on a OSX machine. Is this a typical commercial laptop computer? Does it fail if you start Client() from a different environment? Does it fail if you install tornado==4.5?

My apologies for all of the questions. I would like to track down the source of this issue and, unfortunately, it only appears in a few situations.

[mrocklin]

Also, does this happen after about 30 seconds of hanging around? I would also be interested in seeing the full logs of the following:

from dask.distributed import Client
client = Client(silence_logs=False)

[j-bennet]

Creating a dask-scheduler and dask-workers manually works fine, but creating those by calling Client() with no arguments doesn't work?

Yes.

Does it always fail to work, or does it only fail to work sometimes?

It's intermittent, but it happens more often than not. I would say it happens in 70% of cases.

From your paths it looks like you are on a OSX machine. Is this a typical commercial laptop computer?

It's a MacBook Pro 2013, on High Sierra 10.13.3 (17D102).

Does it fail if you start Client() from a different environment?

I don't have a different environment to test on right now. I'll update the ticket if I do.

Also, does this happen after about 30 seconds of hanging around?

Yes, very much like that.

I'll try tornado==4.5 and verbose logging next.

[mrocklin]

You might also try setting the following in your ~/.dask/config.yaml file

multiprocessing-method: spawn

One reason for the error that you're seeing is that it's taking your system longer than 30s to start a process. Note that I don't mean 30s to start a dask worker and connect to it, I just mean start any standard process. It's is rare to see on unix-based laptop computers (we see this error more often on giant Windows machines or HPC workloads with slow network file sytems), so I'm fairly curious about what might have caused this. There has been some churn recently, both with the recent Tornado 5.0 update and with how we've been adapting to it, and so I'm curious to see which if any of the changes above have some impact.

You might also try https://github.com/dask/distributed/pull/1852

pip install git+https://github.com/mrocklin/distributed@remove-nanny-start-timeout

[j-bennet]

Here is verbose logging:

https://gist.github.com/j-bennet/1c7381cb7ab5e7f896a1547752694c1c

I added 2 log outputs from 2 different runs. Error messages were not quite the same.

[j-bennet]

Downgrading to tornado==4.5 does not solve the issue.

[j-bennet]

Changing to multiprocessing-method: spawn does not help.

Log output: https://gist.github.com/j-bennet/1c7381cb7ab5e7f896a1547752694c1c#file-dask_worker_failed_spawn-log

[mrocklin]

Thanks for taking the trouble to upload those logs @j-bennet . From the looks of it it seems like three of your workers start fine and the four just never shows up, at least not within the 30s alotted to it.

So first question, can you think of any reason why processes might take a long time to start on your machine? As I mentioned earlier this was a more common issue with larger windows boxes or HPC systems, where there some some explanation here. You are, as far as I'm aware, the first person to experience this on an OS-X machine.

You mentioned that things worked nicely when you used the command line interface. Do things start up within a few seconds there, or do they also take tens of seconds to start up?

[j-bennet]

After installing from branch (pip install git+https://github.com/mrocklin/distributed@remove-nanny-start-timeout), this happened again:

https://gist.github.com/j-bennet/1c7381cb7ab5e7f896a1547752694c1c#file-dask_worker_failed_branch-log

but it took 9 runs to happen. So the problem is less frequent, but not completely gone.

[mrocklin]

Hrm, that's actually somewhat surprising. On the runs when it worked how long did it take the process to complete? For reference this usually completes within 1-3 seconds. I'm curious if it completes in more like 20-30 seconds.

In [1]: from dask.distributed import Client

In [2]: %time client = Client()
CPU times: user 499 ms, sys: 28.7 ms, total: 528 ms
Wall time: 1.88 s

[j-bennet]

Yes, it is puzzling to me why things would take this long. My macbook is very well behaved in general. I regularly run Spark jobs on it without any problems. Also, when running Dask or Spark jobs, I usually close all other running applications, because I don't want them interfering with my benchmarks.

When starting workers using the cli, things never hung for too long. Everything was up in a few seconds (I did not time it, but it looked under 5s).

I will get back to you on timing the Client.

[mrocklin]

Ooh, one last question, if it's easy for you to check. Can you reproduce this on Python 3?

I've been repeatedly the remove-nanny-start-timeout branch on travis-ci to see how and where it fails. I've gotten some intermittent failures (not nearly as often as you're able to achieve them, but still), but only on Python 2 so far. Of course it's still entirely possible that this is just the law of small numbers and my personal biases kicking in, but it'd be an interesting experiment to run.

[j-bennet]

When the client does start:

In [3]: %time client = Client()
distributed.diskutils - WARNING - Found stale lock file and directory '/Users/irina/src/talks/2018/daskvsspark/dask-worker-space/worker-YDwfYw', purging
CPU times: user 83.1 ms, sys: 55.2 ms, total: 138 ms
Wall time: 256 ms

In [4]: %time client = Client()
CPU times: user 94.5 ms, sys: 57.8 ms, total: 152 ms
Wall time: 249 ms

In [5]: %time client = Client()
CPU times: user 108 ms, sys: 70.2 ms, total: 178 ms
Wall time: 291 ms

In [6]: %time client = Client()
CPU times: user 119 ms, sys: 81.9 ms, total: 201 ms
Wall time: 267 ms

[j-bennet]

@mrocklin You're definitely on to something with python 3.

I used python3.6.5 and this test script:

from dask.distributed import Client

if __name__ == '__main__':
    client = Client()

I ran this ~15 times and got the following error in most cases (~12 out of 15):

Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 179, in read
    n_frames = yield stream.read_bytes(8)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 179, in read
    n_frames = yield stream.read_bytes(8)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 179, in read
    n_frames = yield stream.read_bytes(8)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/Users/irina/.pyenv/versions/talks3/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed

But this is a different exception! I did not see another "Worker failed to start". Workers start and register just fine.

This new exception only seems to happen when the client exits.

Verbose log:

https://gist.github.com/j-bennet/1c7381cb7ab5e7f896a1547752694c1c#file-dask_worker_error_py3-log

[mrocklin]

Yeah, that error should actually be innocuous. It's due to Tornado's use of asyncio in Tornado 5.0. It should only occur in logs though, and shouldn't affect operation. The fact that things stall only in Python 2 and not 3 is quite interesting. Goes without saying, but thanks for taking the time to run these experiments. I suspect that you'll be happiest with Python 3 and Tornado 4.5 near-term.

[j-bennet]

@mrocklin No problem at all. Is there a way to suppress this exception in logs, since it's not informative anyway?

[mrocklin]

I haven't yet found a nice way to do this yet other than by downgrading to tornado 4.5. Upstream issue here: https://github.com/tornadoweb/tornado/issues/2303

On Tue, Apr 3, 2018 at 11:10 AM, Irina Truong notifications@github.com wrote:

@mrocklin https://github.com/mrocklin No problem at all. Is there a way to suppress this exception in logs, since it's not informative anyway?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1825#issuecomment-378284994, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszKcqt5ITkmIdTVW9OSFEGBwAGlurks5tk5D_gaJpZM4SkVJl .

[mrocklin]

I'm only assuming that that's the same issue. I haven't explored this particular issue too deeply yet.

On Tue, Apr 3, 2018 at 11:13 AM, Matthew Rocklin mrocklin@anaconda.com wrote:

I haven't yet found a nice way to do this yet other than by downgrading to tornado 4.5. Upstream issue here: https://github.com/ tornadoweb/tornado/issues/2303

On Tue, Apr 3, 2018 at 11:10 AM, Irina Truong notifications@github.com wrote:

@mrocklin https://github.com/mrocklin No problem at all. Is there a way to suppress this exception in logs, since it's not informative anyway?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1825#issuecomment-378284994, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszKcqt5ITkmIdTVW9OSFEGBwAGlurks5tk5D_gaJpZM4SkVJl .

[mrocklin]

The problem is here in client.py:899

try:
    msgs = yield self.scheduler_comm.comm.read()
except CommClosedError:
    handle exceptions sensibly

We are reading on an IOStream and, if it breaks, we handle the break. However when the IOLoop shuts down the exception handling code doesn't get a chance to deal with things, and asyncio emits the error and I'm not familiar enough with that codebase to know how to control its verbosity. This is a break from how Tornado used to handle these situations until the recent switch.

[mrocklin]

@bdarnell might have some suggestions, though he may also be busy

[bdarnell]

Why is the IOLoop shutting down? If you have cleanup work that must happen, it's up to the application to not stop the IOLoop until that work is done.

Shutting down an IOLoop is in some sense very well-tested (since every unit test does it), but it's also very unusual and under-tested in practice. Personally I never call IOLoop.stop from non-test code in systems I build; I just use os.exit() once I've finished my draining process.

[mrocklin]

Why is the IOLoop shutting down?

In @j-bennet 's case (and many others) they're shutting down a local or interactive session. In other cases they might spin up dask only temporarily with the IOLoop living in a separate thread, connect to an external cluster, and then shut everything down when they're done. They might do this several times within the lifetime of the process.

If you have cleanup work that must happen, it's up to the application to not stop the IOLoop until that work is done.

We clean most things up, but not everything. This might be an opportunity for improvement.

I just use os.exit() once I've finished my draining process

That's an interesting option. I can confirm that just not calling IOLoop.stop when we're shutting down the process anyway resolves the problem.

[mrocklin]

Quick and dirty solution implemented here: https://github.com/dask/distributed/pull/1895

[mrocklin]

Some of the "Worker failed to start" issues on Python 2 seem to be somewhat resolved with https://github.com/dask/distributed/pull/1905

[mrocklin]

Though not entirely

[allentsouhuang]

I ran into this issue today and I have a 100% reproducible case as well as a possible explanation:

If I use the spawn start_method for multiprocessing in python 3.6.7, the following code returns normally:

from dask.distributed import Client
c = Client()

By keeping the default forkserver mode, dask is failing to spawn any workers since forking on macOS is very bad. (I see python crash files in ~/Library/Logs/DiagnosticReports).

However, it looks like multiprocessing start methods were added in python 3.4.

If I use python2.7 and keep the .dask/config.yaml to use 'spawn', the code snippet above just hangs. I think this is because you can't set the start method of the multiprocessing module under python 3.4.

[allentsouhuang]

As a result, LocalCluster with multiprocessing is just broken on macOS running python2.7. =/

[PGryllos]

@here I am facing a similar issue with Client() on python2.7; the cluster sometimes starts and sometimes doesn't. Not sure what causes it to fail ftm.

[PGryllos]

when trying with

from dask.distributed import Client
client = Client('localhost:8789')

I get

IOError: Timed out trying to connect to 'tcp://localhost:8789' after 10 s: in <distributed.comm.tcp.TCPConnector object at 0x11c577890>: error: [Errno 61] Connection refused

[PGryllos]

even with something like

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(
     n_workers=4,
     ip='127.0.0.1',
)
client = Client(cluster)

the cluster is not guaranteed to start :/ is there a workaround to make sure that the cluster will always start?

[mrocklin]

I'm curious, you specifically mentioned that you were on Python 2.7. Does this not occur on Python 3?

How about if you pass processes=False?

[mrocklin]

Does this occur if you start things on the command line with dask-scheduler and dask-worker?

[allentsouhuang]

If I pass processes=False, then it works just fine. If I run desk-scheduler and desk-worker in the command line and connect to it via Client in both python2 and python3, it works just fine.

I believe this has to do with how processes work on macOS. If a process uses the libdispatch library for asynchronous work, the OS marks it as a multi-threaded process complete with an objective C runtime. A process with an objective C runtime under the hood can NOT be forked (i.e. it crashes)

So my theory is that if a python process uses any threading (implemented under the hood with libdispatch) prior to forking, it will crash.

Starting in python3, you can specify whether to spawn a fresh new python process which circumvents the issue or forkserver (the default case).

So to reiterate:

python3 + "spawn" + LocalCluster()=> success python3 + "forkserver" + LocalCluster() => fail python2 + LocalCluster() => fail

python3 + "forkserver" + LocalCluster(processes=False) => success python3 + "spawn" + LocalCluster(processes=False) => success python2 + LocalCluster(processes=False) => success

Given that my workload is cpu bound and running python2, using a thread pool over a process pool won't give me the speedup that I'm looking for.

On a related note: This issue can be pretty subtle. I first ran into this issue while using the requests library. In one of the comments of https://stackoverflow.com/questions/28521535/requests-how-to-disable-bypass-proxy, it says that requests will check to see if the system has configured any proxies, which requires the python process to communicate with cfprefsd which then marks it as a multi-threaded environment. Then if you try to fork the python process, then it will crash.

[ilpomo]

PyCharm 2018.3.2, Windows 10 Pro 64bit.

from dask.distributed import Client

if __name__ == '__main__':
    c = Client()
    print(c)

Running the code above in Python Interactive Console returns this error.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 14416 exited with status 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
distributed.nanny - WARNING - Worker process 12624 exited with status 1
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 14612 exited with status 1
distributed.nanny - WARNING - Worker process 8184 exited with status 1
distributed.nanny - WARNING - Worker process 6424 exited with status 1
distributed.nanny - WARNING - Worker process 2408 exited with status 1
distributed.nanny - WARNING - Worker process 9016 exited with status 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
distributed.nanny - WARNING - Worker process 9324 exited with status 1
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
distributed.nanny - WARNING - Worker process 8320 exited with status 1
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 7264 exited with status 1
distributed.nanny - WARNING - Worker process 15036 exited with status 1
distributed.nanny - WARNING - Worker process 704 exited with status 1
distributed.nanny - WARNING - Worker process 3660 exited with status 1
distributed.nanny - WARNING - Worker process 15952 exited with status 1
distributed.nanny - WARNING - Worker process 3088 exited with status 1
distributed.nanny - WARNING - Worker process 3996 exited with status 1
distributed.nanny - WARNING - Worker process 3512 exited with status 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 14280 exited with status 1
distributed.nanny - WARNING - Worker process 4700 exited with status 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 11736 exited with status 1
distributed.nanny - WARNING - Worker process 8988 exited with status 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 7952 exited with status 1
distributed.nanny - WARNING - Worker process 4412 exited with status 1
distributed.nanny - WARNING - Worker process 11380 exited with status 1
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 8052 exited with status 1
distributed.nanny - WARNING - Worker process 7616 exited with status 1
distributed.nanny - WARNING - Worker process 8976 exited with status 1
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 13812 exited with status 1
distributed.nanny - WARNING - Worker process 2812 exited with status 1
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
    exitcode = _main(fd)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
    run_name="__mp_main__")
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file
    with open(fname, "rb") as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\thoma\\My Tresors\\Private\\midori\\<input>'
distributed.nanny - WARNING - Worker process 15220 exited with status 1
distributed.nanny - WARNING - Worker process 15920 exited with status 1
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
Traceback (most recent call last):
  File "<input>", line 6, in <module>
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 636, in __init__
    self.start(timeout=timeout)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 759, in start
    sync(self.loop, self._start, **kwargs)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\utils.py", line 277, in sync
    six.reraise(*error[0])
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\utils.py", line 262, in f
    result[0] = yield future
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run
    value = future.result()
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 822, in _start
    yield self.cluster
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run
    value = future.result()
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\asyncio\tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 191, in _start
    yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run
    value = future.result()
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback
    result_list.append(f.result())
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start

Any solution?

[mrocklin]

I'm curious what the runpy.py file is. Is that something particular to your environment perhaps?

On Fri, Jan 11, 2019 at 3:44 PM ilpomo notifications@github.com wrote:

PyCharm 2018.3.2, Windows 10 Pro 64bit.

from dask.distributed import Client

if name == 'main': c = Client() print(c)

Running the code above in Python Interactive Console returns this error.

Traceback (most recent call last): File "", line 1, in Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main Traceback (most recent call last): File "", line 1, in prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 14416 exited with status 1 Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare distributed.nanny - WARNING - Worker process 12624 exited with status 1 _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 14612 exited with status 1 distributed.nanny - WARNING - Worker process 8184 exited with status 1 distributed.nanny - WARNING - Worker process 6424 exited with status 1 distributed.nanny - WARNING - Worker process 2408 exited with status 1 distributed.nanny - WARNING - Worker process 9016 exited with status 1 Traceback (most recent call last): File "", line 1, in distributed.nanny - WARNING - Worker process 9324 exited with status 1 File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file distributed.nanny - WARNING - Worker process 8320 exited with status 1 with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 7264 exited with status 1 distributed.nanny - WARNING - Worker process 15036 exited with status 1 distributed.nanny - WARNING - Worker process 704 exited with status 1 distributed.nanny - WARNING - Worker process 3660 exited with status 1 distributed.nanny - WARNING - Worker process 15952 exited with status 1 distributed.nanny - WARNING - Worker process 3088 exited with status 1 distributed.nanny - WARNING - Worker process 3996 exited with status 1 distributed.nanny - WARNING - Worker process 3512 exited with status 1 Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 14280 exited with status 1 distributed.nanny - WARNING - Worker process 4700 exited with status 1 Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 11736 exited with status 1 distributed.nanny - WARNING - Worker process 8988 exited with status 1 Traceback (most recent call last): File "", line 1, in Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 7952 exited with status 1 distributed.nanny - WARNING - Worker process 4412 exited with status 1 distributed.nanny - WARNING - Worker process 11380 exited with status 1 distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Restarting worker Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 8052 exited with status 1 distributed.nanny - WARNING - Worker process 7616 exited with status 1 distributed.nanny - WARNING - Worker process 8976 exited with status 1 distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Restarting worker Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare Traceback (most recent call last): File "", line 1, in _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 13812 exited with status 1 distributed.nanny - WARNING - Worker process 2812 exited with status 1 distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Restarting worker Traceback (most recent call last): File "", line 1, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main Traceback (most recent call last): File "", line 1, in prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 105, in spawn_main File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") exitcode = _main(fd) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 225, in prepare code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\thoma\Anaconda3\envs\midori\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' run_name="mp_main") File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 261, in run_path code, fname = _get_code_from_file(run_name, path_name) File "C:\Users\thoma\Anaconda3\envs\midori\lib\runpy.py", line 231, in _get_code_from_file with open(fname, "rb") as f: OSError: [Errno 22] Invalid argument: 'C:\Users\thoma\My Tresors\Private\midori\' distributed.nanny - WARNING - Worker process 15220 exited with status 1 distributed.nanny - WARNING - Worker process 15920 exited with status 1 tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start Traceback (most recent call last): File "", line 6, in File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 636, in init__ self.start(timeout=timeout) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 759, in start sync(self.loop, self._start, *kwargs) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\utils.py", line 277, in sync six.reraise(error[0]) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\six.py", line 693, in reraise raise value File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\utils.py", line 262, in f result[0] = yield future File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run value = future.result() File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1141, in run yielded = self.gen.throw(exc_info) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\client.py", line 822, in _start yield self.cluster File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run value = future.result() File "C:\Users\thoma\Anaconda3\envs\midori\lib\asyncio\tasks.py", line 537, in _wrap_awaitable return (yield from awaitable.await()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1141, in run yielded = self.gen.throw(exc_info) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 191, in _start yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)] File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1133, in run value = future.result() File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 883, in callback result_list.append(f.result()) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\tornado\gen.py", line 1147, in run yielded = self.gen.send(value) File "C:\Users\thoma\Anaconda3\envs\midori\lib\site-packages\distributed\deploy\local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start

Any solution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1825#issuecomment-453692658, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNJCtdVgysWXFhywkMc56h51CgAPks5vCSHCgaJpZM4SkVJl .

[ilpomo]

@mrocklin I've created my conda environment through the normal PyCharm UI, from a fresh Anaconda installation. Here you can download the unmodified file runpy.py.

Could this error be caused by the multiprocessing on the Python interactive shell?

[mrocklin]

I recommend trying to run that file from a normal Python executable by calling python.exe myscript.py and see if that also fails. If not then probably PyCharm is doing something odd.

On Fri, Jan 11, 2019 at 4:37 PM ilpomo notifications@github.com wrote:

@mrocklin https://github.com/mrocklin I've created my conda environment through the normal PyCharm UI, from a fresh Anaconda installation. Here you can download the unmodified file runpy.py. https://ufile.io/jo5z8

Could this error be caused by the multiprocessing on the Python interactive shell?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1825#issuecomment-453701621, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszHZE2koHq4XrXyS0xiJuNDs7k7bMks5vCS5OgaJpZM4SkVJl .

[ilpomo]

@mrocklin I tried creating a new conda environment, same result. Then I tried using my local Python interpreter as interpreter for the project and now everything works. I don't understand what is wrong with conda environments.

Btw I would like to thank you for your work on Dask and please, don't remove actors. Without them, it becomes really hard to conceptualize a software structure only based on functions that is similar to what the in production reality is.

[mrocklin]

@mrocklin I tried creating a new conda environment, same result. Then I tried using my local Python interpreter as interpreter for the project and now everything works. I don't understand what is wrong with conda environments.

I encourage you to try running python.exe in a command line rather than using PyCharm to see if that has an effect. If so, then I encourage you to report upstream to PyCharm. I suspect that they're doing something odd with launching Python scripts.

Actually. it looks like this was reported a while ago here: https://github.com/dask/distributed/issues/2139

[victor-ab]

I have the same issue as @ilpomo here using python 3.6.2, dask 1.1.5 and distributed 1.26.1. The funny thing is that it works properly outside jupyter notebook when I run it directly on ipython on bash. Another info is that it also worked within python 2.7. Although I was not able to get the cli dask-scheduler neither the dask-manager, it was only sucessfull when using the python api.

[Athlete369]

from dask.distributed import Client, progress client = Client(n_workers=2, threads_per_worker=2, memory_limit='1GB') client this is giving me error on linux

[TomAugspurger]

@Athlete369 can you provide more information? What version of distributed? What's the traceback?

[Athlete369]

https://github.com/dask/distributed/issues/1825#issuecomment-501879240 Tornador.util timeout error