iseessel
commented
2 years ago Hi @robin-karlsson0 thanks for reaching out and glad to hear your using vissl for large-scale distributed training!
node_id should be set in the configuration here https://github.com/facebookresearch/vissl/blob/main/vissl/config/defaults.yaml#L49. It then gets passed in here: https://github.com/facebookresearch/vissl/blob/main/vissl/utils/distributed_launcher.py#L134 and finally the distributed rank gets set here: https://github.com/facebookresearch/vissl/blob/main/vissl/utils/distributed_launcher.py#L182.
I am not familiar with your setup, but the easiest fix would be if somehow you could retrieve the NODE_ID through an environment variable in your script. You can see for example how we get the env variable for SLURM: https://github.com/facebookresearch/vissl/blob/main/vissl/utils/slurm.py#L11, there should be something similar for your system.
config.node_id=$NODE_ID_ENV_VARIABLEYou also have to specify the following for your training since you want to use mpi, 2 nodes and 4 gpus per node. I noticed you load nccl -- if your setup supports nccl training, I would recommend using NCCL instead. It should be faster and is recommended by pytorch. Vissl is also not well-tested on an mpi environment.
config.DISTRIBUTED.BACKEND=mpi
config.DISTRIBUTED.NUM_NODES=2 \
config.DISTRIBUTED.NUM_PROC_PER_NODE=4 \And finally, I would recommend altering the sync batchnorm groupsize from 8 to 4. This will sync batchnorm mean and variance statistics across one node, which should improve performance as it doesn't require across node communication.
config.MODEL.SYNC_BN_CONFIG.GROUP_SIZE=4So your final script would be something like:
#!/bin/bash -x
#PJM -L rscgrp=####
#PJM -L node=2
#PJM -L elapse=5:00
#PJM -j
#PJM -S
module load gcc/8.4.0
module load python/3.8.11
module load cuda/11.2.1
module load cudnn/8.1.1
module load nccl/2.8.4
module load openmpi_cuda/4.0.5
source /home/###/.pyenv/versions/vissl/bin/activate
# distributed setting
MASTER_ADDR=$(head -n 1 ${PJM_O_NODEINF})
mpirun -np 8 -machinefile ${PJM_O_NODEINF} -map-by ppr:4:node \
python tools/run_distributed_engines.py \
config=pretrain/swav/swav_8node_resnet.yaml \
config.DISTRIBUTED.RUN_ID="${MASTER_ADDR}":29500 \
config.DISTRIBUTED.NUM_NODES=2 \
config.DISTRIBUTED.NUM_PROC_PER_NODE=4 \
config.MODEL.SYNC_BN_CONFIG.GROUP_SIZE=4 \
config.DISTRIBUTED.BACKEND=mpi \
config.node_id=$NODE_ID_ENV_VARIABLELMK if that makes sense and/or if you have any questions!
robin-karlsson0
Hi. Does anyone know the correct way to launch VISSL jobs using MPI on a supercomputer with multiple nodes/machines without SLURM?
Use case: 2 nodes w. 4 GPUs each
As far as I understand, the issue seems to be that instead of a single master node w. rank 0, 8 separate rank 0 nodes and 4 processes seem to be spawned. Is my launch script incorrect? Is there some conflict between the MPI and NCCL backends?
Hope someone knows an easy fix or correction to my configuration! Thank you :slightly_smiling_face:
...
Traceback (most recent call last): File "tools/run_distributed_engines.py", line 61, in
hydra_main(overrides=overrides)
File "tools/run_distributed_engines.py", line 44, in hydra_main
launch_distributed(
File "/home/z44406a/projects/vissl/vissl/utils/distributed_launcher.py", line 135, in launch_distributed
torch.multiprocessing.spawn(
File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/home/z44406a/projects/vissl/vissl/utils/distributed_launcher.py", line 192, in _distributed_worker run_engine( File "/home/z44406a/projects/vissl/vissl/engines/engine_registry.py", line 86, in run_engine engine.run_engine( File "/home/z44406a/projects/vissl/vissl/engines/train.py", line 39, in run_engine train_main( File "/home/z44406a/projects/vissl/vissl/engines/train.py", line 127, in train_main trainer = SelfSupervisionTrainer( File "/home/z44406a/projects/vissl/vissl/trainer/trainer_main.py", line 85, in init self.setup_distributed(self.cfg.MACHINE.DEVICE == "gpu") File "/home/z44406a/projects/vissl/vissl/trainer/trainer_main.py", line 117, in setup_distributed torch.distributed.init_process_group( File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 534, in init_process_group store, rank, world_size = next(rendezvous_iterator) File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 184, in _tcp_rendezvous_handler store = _create_c10d_store(result.hostname, result.port, rank, world_size, timeout) File "/home/z44406a/.pyenv/versions/3.8.11/envs/vissl/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 158, in _create_c10d_store return TCPStore( RuntimeError: Address already in use
!/bin/bash -x
PJM -L rscgrp=
PJM -L node=2
PJM -L elapse=5:00
PJM -j
PJM -S
module load gcc/8.4.0 module load python/3.8.11 module load cuda/11.2.1 module load cudnn/8.1.1 module load nccl/2.8.4 module load openmpi_cuda/4.0.5
source /home/###/.pyenv/versions/vissl/bin/activate
distributed setting
MASTER_ADDR=$(head -n 1 ${PJM_O_NODEINF})
mpirun -np 8 -machinefile ${PJM_O_NODEINF} -map-by ppr:4:node \ python tools/run_distributed_engines.py \ config=pretrain/swav/swav_8node_resnet.yaml \ config.DISTRIBUTED.RUN_ID="${MASTER_ADDR}":29500
DISTRIBUTED: BACKEND: nccl NUM_NODES: 2 # user sets this to number of machines to use NUM_PROC_PER_NODE: 4 # user sets this to number of gpus to use per machine INIT_METHOD: tcp # recommended if feasible otherwise RUN_ID: localhost:{port} # select the free port