Open FANLONGFANLONG opened 3 years ago
What happened:
we used the below way to start dask cluster with 20 workers
n_workers = 20 spec_worker = skein.Service(resources=skein.Resources(memory=memory_limit, vcores=threads_per_worker), instances=n_workers, files=resource_files, script=worker_script) services["dask.worker"] = spec_worker spec_scheduler = skein.Service(resources=skein.Resources(memory="20g", vcores=4), files=resource_files, instances=1, script=scheduler_script) services["dask.scheduler"] = spec_scheduler spec = skein.ApplicationSpec(name=app_name, queue=_get_queue(), master=spec_master, services=services) cluster = YarnCluster.from_specification(spec)
What you expected to happen: all of 20 workers registered into scheduler
Anything else we need to know?:
Environment:
This seems to be related to how dask-yarn exposes scheduler logging so moving the issue over there.
dask-yarn
What happened:
we used the below way to start dask cluster with 20 workers
What you expected to happen: all of 20 workers registered into scheduler
Anything else we need to know?:
Environment: