`Runtime error : replicas ...` in distributed training

[zsef123]

Describe the bug In distributed train, raised error at evaluation step. And working fine in single gpu mode

Reproduction

1. What command or script did you run?

   CUDA_VISIBLE_DEVICES=0,1,2,3 ./tools/dist_train.sh \
   configs/fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_4x4_1x_coco.py \
   4 --work-dir multi_gpu_test

2. Did you make any modifications on the code or config? Did you understand what you have modified?

   • I changed the dataset to fit my set
   • But running fine in python3 tools/train.py ....

3. What dataset did you use?

   • COCO Like

Environment

➜ python3 mmdet/utils/collect_env.py
sys.platform: linux
Python: 3.7.8 (default, Jun 29 2020, 05:44:46) [GCC 7.5.0]
CUDA available: True
GPU 0,1,2,3: Tesla V100-SXM2-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.7.0+cu101
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
  - CuDNN 7.6.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.1+cu101
OpenCV: 4.3.0
MMCV: 1.1.6
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 10.1
MMDetection: 2.5.0+eb7bfbc

Error traceback

• removes duplicated error trace

  Traceback (most recent call last):
  File "./tools/train.py", line 178, in <module>
  main()
  File "./tools/train.py", line 174, in main
  meta=meta)
  File "/data2/DW/200619_blood_cell/blood_analyzer/projects/cbc_2d/mmdetection/mmdet/apis/train.py", line 150, in train_detector
  runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/user/.local/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 126, in run
  epoch_runner(data_loaders[i], **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 55, in train
  self.call_hook('after_train_epoch')
  File "/home/user/.local/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
  getattr(hook, fn_name)(self)
  File "/data2/DW/200619_blood_cell/blood_analyzer/projects/cbc_2d/mmdetection/mmdet/core/evaluation/eval_hooks.py", line 129, in after_train_epoch
  gpu_collect=self.gpu_collect)
  File "/data2/DW/200619_blood_cell/blood_analyzer/projects/cbc_2d/mmdetection/mmdet/apis/test.py", line 96, in multi_gpu_test
  result = model(return_loss=False, rescale=True, **data)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
  result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/distributed.py", line 606, in forward
  if self.reducer._rebuild_buckets():
  RuntimeError: replicas_[0].size() == rebuilt_param_indices_.size() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/distributed/c10d/reducer.cpp":1326, please report a bug to PyTorch. rebuilt parameter indices size is not same as original model parameters size.218 versus 654

[zsef123]

Same issue with https://github.com/pytorch/pytorch/issues/47050

[PLester]

！！！ remember set following in you cfg file: find_unused_parameters = True