Open tanzir5 opened 4 months ago
srun --mpi-pmi2 python script.py
worked with 1 GPU on 19/04.It seems multi-gpu will work with the MPI backend, but it's unclear how much the CPU-GPU transfer slows down training.
Interestingly the above command also works for multi-gpu training for the dummy model but it fails for the Language model, regardless of the number of gpus. I should verify on Snellius that the language model can train using DDP over MPI, and also maybe on Stony Brook server once we have fake data.
see also #23
An intermediate conclusion from #23 is to use srun
, but not MPI backend. The current code is about as fast with nccl and gloo backends. We have not tried either on the OSSC though. Depending on the outcome of #74, we need to make nccl work on the OSSC as well.
by now I know that NCCL works on the OSSC with pytorch (with FSDP on a much larger model than ours). We need to figure out if our issue persists and if it's down to lightning or to our own code.
Hey I tried with some PyTorch DDP (using NCCL)
I think we need to set
NCCL_SOCKET_IFNAME=lo
in the jobscript
Actually probably better
export NCCL_SOCKET_IFNAME=ib0
You are right that NCCL does work, it just seems that we need to be explicit in the OSSC
Summarizing the situation here (Apr 16):
TODO FRIDAY (APR 19):