awaelchli
commented
2 years ago Hi @casparvl This is the implementation of _is_slurm_managing_tasks:
def _is_slurm_managing_tasks(self) -> bool:
"""used by choosing cluster enviroment."""
if not SLURMEnvironment.detect() or SLURMEnvironment.job_name() == "bash":
return False
total_requested_devices = len(self._parallel_devices) * self._num_nodes_flag
num_slurm_tasks = int(os.environ["SLURM_NTASKS"], 0)
return num_slurm_tasks == total_requested_devicesAnd yes, it needs to return True for your job to be launched correctly. Can you tell me which of these conditions return the wrong value? Please print the following values:
isinstance(trainer.strategy.cluster_environment, SLURMEnvironment) # True
SLURMEnvironment.detect() # True
SLURMEnvironment.job_name() # should not be "bash"
os.environ["SLURM_NTASKS"] # 4
len(trainer.accelerator_connector._parallel_devices) # 4Also, set
trainer = Trainer(
...,
accelerator='gpu',
devices=4
strategy='ddp',
)maybe the gpus=-1 is the problem.
casparvl
🐛 Bug && steps to reproduce
I'm running the standard
boring_model(or any model), with some minor changes to the arguments:on our SLURM cluster like so:
however, it seems to spawns 8 processes: 4 'worlds' of size two. While running, each GPU is running two processes. I also don't see "Multiprocessing is handled by SLURM" in the output, which is expected from here.
Expected behavior
I expect
srunto launch 4 tasks, and PyTorch Lightnings SLURMEnvironment to make these into a single world of size 4.Environment
Additional context
The issue first poped up in a real-world code where I was combining PT Lightning with Hydra, and showed up as a rather strange error message from Hydra. This thread contains a lot more analysis on that. However, I'll repeat/summarize the essentials here.
The problem arrises because
_is_slurm_managing_tasks(self)inaccelerator_connector.pyis returningFalse, despite the tasks being launched by SLURM. The result is that PyTorch Lightning will itself try to spawn tasks, from each of the four processes already launched bysrun. As we'll see later, each process sees 2 GPUs, and thus launches two tasks, thus I end up with my final total of 8 tasks.Since PyTorch Lightning thinks it's running outside of a SLURM context, that also causes the hydra error. This is because Lightning will use
subprocess_script.pyto try and launch a new process. Here it will addhydra.run.dirto the original argument list and call the original command again, but since the original command is the_submit.pyscript (and not the PyTorch lightning training script) this fails in an unrecognized argument error.That part aside, the real problem is: why doesn't Lightning recognize that slurm is managing the tasks? Well, that's because of this line. Lightning assumes two things here:
self._parallel_devicescontains the number of physical parallel devices that is available per nodeI won't go deeper into number (2), since that assumption isn't broken here - I just want to remark that
_is_slurm_managing_tasks()logic would also break in that case. In our case, the problem is (1).Let's see what's going on step by step.
self._parallel_devicesis set here, based on a call toself.accelerator.get_parallel_devices(self._devices_flag). In turn, theself._devices_flagis set here in my case, based on a call toself.accelerator.auto_device_count(). Since my accelerator is a GPU,self.acceleratoris aGPUAcceleratorobject. Thus, it's calling this function. And this, suprisingly, is where the problem lies.auto_device_countwill return the number of devices that that process has access to. That is potentially not the same as the number of _physical parallel devices that is available per node` in a SLURM allocation.To demonstrate, consider the following run:
As you can see, SLURM launches 4 tasks. Each of those tasks get access to a subset of the devices available on the node. In this case,
auto_device_countwill return 2 - the number of GPUs that that particular process has access to. The reason it only has access to two is because of the--gpu-bind=closestargument (see the documentation of srun). This will make sure that GPUs are bound to each task that are closest (in a NUMA-sense) to the CPU processes controlling them. To demonstrate:As you can see, each task gets bound to 18 cores. This system has 36 cores per socket, and 2 sockets in total. SLURM binds task 0 to CPUs 0-17 on Socket 0 and sets
CUDA_VISIBLE_DEVICES=0,1for that task, because those two GPUs are attached to the PCI bridge of that socket. Similarly, it binds task 1 to CPUs 18-35 on Socket 0 and also setsCUDA_VISIBLE_DEVICES=0,1(because those CPUs are still on Socket 0, so these are attached to that PCI bridge). Then, it binds task 2 to CPUs 36-53 on Socket 1, and setCUDA_VISIBLE_DEVICES=2,3, and it binds task 3 to CPUs 54-71 on Socket 1, and setsCUDA_VISIBLE_DEVICES=2,3. Please note that all of this is completely intended and correct SLURM behaviour. Thus, ideally, theSLURMEnvironmentshould be able to deal with it - yet it isn't.I have tried to just hard-code the output of
_is_slurm_managing_taskstoTrue, just to see if that would fix it. However, this just simply produces more errors: theSLURMEnvironmentis now being used, but it results inprobably because the
self._devices_flagis set to[0,1]for each task, even though two out of the tasks only have access to devices[2,3].I'm not really sure how to 'fix' this, it would require changes both in the
_is_slurm_managing_tasksfunction, but also in the way thatSLURMEnvironmentassigns devices to each of the processes...cc @awaelchli @akihironitta