search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.58k stars 718 forks source link

LocalCluster in processes=False doesnt accept the keyword worker_class #2447

Open redsum opened 5 years ago

redsum commented 5 years ago

this will fail

from distributed import Worker
cluster = LocalCluster(processes=False, worker_class=Worker)

while this will pass

from distributed import Worker
cluster = LocalCluster(processes=True, worker_class=Worker)
mrocklin commented 5 years ago

Yes, the processes= keyword is really just syntactic sugar around using the Nanny rather than the Worker class.

What is it that you're trying to achieve?

redsum commented 5 years ago

I'm using multiprocessing, and i want to have a custom initialization for each process So i use the c'tor of the worker to perform the initialization

But the main point here is that There shouldn't be a difference in API of multiprocessing and Multithreading