SamTov
commented
8 months ago My best solution at the moment to force this to do what I want it to is to have a function like the following:
def deploy_jobs(index):
cluster = SLURMCluster(
cores=16,
processes=1,
memory="64GB",
queue="single",
walltime="03:00:00",
death_timeout="15s",
worker_extra_args=["--resources GPU=1"],
log_directory=f'./ce-two-layer-10/dask-logs',
job_script_prologue=["module load devel/cuda/12.1"],
job_extra_directives=["--gres=gpu:1"]
)
cluster.scale(1)
client = Client(cluster)
return client.submit(
train, index, resources={"GPU": 1}
)I then loop over my parameters, create a cluster, scale it, and submit jobs to it. Essentially wrapping the submit function of Dask with a short cluster setup step. Not ideal, but it works perfectly.
The obvious downside is that I can't collect the resource information into a single dashboard to monitor.
KonstiNik
guillaumeeb
Describe the issue: This is likely a misunderstanding of how to correctly use Dask to deploy cluster jobs. However, the terminology in the documentation suggests what should be happening, so I also see this as a type of bug as the functionality is so different to what one might expect.
I am trying to train a large number of machine learning models on a SLURM cluster. Each Node has 64 cores and 4 GPUs. I want to run each of my model with 1 GPU and 16 cores so I can, theoretically, get 4 models on each Node and maximise my resources.
My input script is summarised as follows:
My expected behaviour is that Dask submits five workers to the queue. Each worker takes a network to train with a given index, trains it on 16 cores and 1 GPU, and starts the next one when the training is finished. What happens, however, is that four workers are submitted to the queue, and only one of them starts to take networks and train them sequentially. The other workers are just idle.
I have tried increasing the number of processes, which I would think means running multiple network trainings on a single worker and splitting the resources. But this is also not correct as, in this case, it gives each process its own GPU despite the worker theoretically only having access to one. It also only runs on a single worker; the others are left idling.
I have also tried using map instead of submit. In this case, the workers die, or they try to run as many network trainings as possible on a single worker.
Finally, I have also tried using adapt, which is preferential to my workflow. However, when I do so, all of my workers keep dying with no logs produced in an endless cycle.
Even though I am reasonably familiar with clusters, especially SLURM clusters, as I mentioned above, I think I am missing something about how the API is supposed to work.