Closed SSSHZ closed 2 years ago
Hello, I think this bug is caused by allocating too many workers for dataloader. When training with multi gpus, ${--bs} is the number of batch size of a single gpu, the actual batch size of your above command is 24*4=96. For convenience, I directly set the number of workers as the number of batch size. 96 workers for dataloder is probably too many.
You can try set the batch size smaller, such as --bs 6 when using 4 GPUS. Or you can modify num_worker of the function make_ddp_train_loader in dataset/data_loader.py.
After setting train.batch_size = 6
in configs/coco.py
, I tried num_workers=4 or 2 or 0
for make_ddp_train_loader
in dataset/data_loader.py
and the same issue always happened.
Perhaps this bug is not caused by workers for dataloader.
The problem might be caused by the combination of Pytorch 1.7.1, CUDA 10.2, NCCL 2.7.8
.
The easiest solution is to try Pytorch 1.7.0
for me.
Hi, I tried the multi-gpu training code but the program always got stuck after a few iterations.
Environment:
Reproduce the bug:
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node 4 train_net_ddp.py --config_file coco --gpus 4
Output: