AttributeError: 'MMDistributedDataParallel' object has no attribute '_use_replicated_tensor_module'

[xuch98]

something is wrong when I execute the command below for training the model on my own dataset. bash ./dist_train.sh configs/mask_rcnn_efficientvit_m4_fpn_1x_coco.py 4 --cfg-options model.backbone.pretrained=./runs/efficientvit_m4.pth What I have done is just formatting my dataset into COCO-type and downloading the pretrained checkpoint. Here is the detailed information:

2023-06-12 20:57:08,304 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs 2023-06-12 20:57:08,304 - mmdet - INFO - Checkpoints will be saved to /home/xc/transform/Cream/EfficientViT/downstream/work_dirs/mask_rcnn_efficientvit_m4_fpn_1x_coco by HardDiskBackend. 2023-06-12 20:57:10,480 - mmdet - INFO - Saving checkpoint at 1 epochs [ ] 0/81, elapsed: 0s, ETA:Traceback (most recent call last): File "/home/xc/transform/Cream/EfficientViT/downstream/./train.py", line 245, in main() File "/home/xc/transform/Cream/EfficientViT/downstream/./train.py", line 234, in main train_detector( File "/home/xc/transform/Cream/EfficientViT/downstream/mmdet_custom/apis/train.py", line 184, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run epoch_runner(data_loaders[i], kwargs) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/runner/epoch_based_runner.py", line 58, in train self.call_hook('after_train_epoch') File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook getattr(hook, fn_name)(self) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/runner/hooks/evaluation.py", line 271, in after_train_epoch self._do_evaluate(runner) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmdet/core/evaluation/eval_hooks.py", line 126, in _do_evaluate results = multi_gpu_test( File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmdet/apis/test.py", line 109, in multi_gpu_test result = model(return_loss=False, rescale=True, data) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1535, in forward else self._run_ddp_forward(inputs, **kwargs) File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/parallel/distributed.py", line 160, in _run_ddp_forward self._use_replicated_tensor_module else self.module File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'MMDistributedDataParallel' object has no attribute '_use_replicated_tensor_module'

[xinyuliu-jeffrey]

Hi @xuch98 ,

This seems to be related to the mmcv package version or torch version. Could you provide your package versions in pip list? Thanks.

[xuch98]

@xinyuliu-jeffrey Thank you for your reply The version of CUDA is 11.7 $ pip list Package Version Editable project location

---

absl-py 1.4.0 addict 2.4.0 apex 0.1 beautifulsoup4 4.12.2 bs4 0.0.1 cachetools 5.3.0 certifi 2022.12.7 charset-normalizer 2.1.1 click 8.1.3 colorama 0.4.6 coloredlogs 15.0.1 contourpy 1.0.7 cycler 0.11.0 filelock 3.9.0 flatbuffers 23.3.3 fonttools 4.39.3 fsspec 2023.5.0 gitdb 4.0.10 GitPython 3.1.31 google-auth 2.17.3 google-auth-oauthlib 1.0.0 grpcio 1.53.0 huggingface-hub 0.14.1 humanfriendly 10.0 idna 3.4 imageio 2.28.1 Jinja2 3.1.2 kiwisolver 1.4.4 lazy_loader 0.2 Markdown 3.4.3 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib 3.7.1 mdurl 0.1.2 mmcv-full 1.7.1 mmdet 2.27.0 mmengine 0.7.3 model-index 0.1.11 mpmath 1.2.1 networkx 3.0rc1 numpy 1.24.1 oauthlib 3.2.2 onnx 1.13.1 onnxruntime 1.14.1 opencv-python 4.7.0.72 openmim 0.3.7 ordered-set 4.1.0 packaging 23.1 pandas 2.0.0 Pillow 9.3.0 pip 23.1.2 pip-search 0.0.12 protobuf 3.20.3 psutil 5.9.4 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycocotools 2.0.6 Pygments 2.15.1 pyparsing 3.0.9 python-dateutil 2.8.2 pytorch-triton 2.1.0+46672772b4 pytz 2023.3 PyWavelets 1.4.1 PyYAML 6.0 requests 2.28.1 requests-oauthlib 1.3.1 rich 13.3.5 rsa 4.9 safetensors 0.3.1 scikit-image 0.20.0 scipy 1.10.1 seaborn 0.12.2 segment-anything 1.0 /home/xc/transform/segment-anything setuptools 65.6.3 shapely 2.0.1 six 1.16.0 smmap 5.0.0 soupsieve 2.4.1 sympy 1.11.1 tabulate 0.9.0 tensorboard 2.12.2 tensorboard-data-server 0.7.0 tensorboard-plugin-wit 1.8.1 termcolor 2.3.0 terminaltables 3.1.10 thop 0.1.1.post2209072238 tifffile 2023.4.12 timm 0.9.2 tomli 2.0.1 torch 2.1.0.dev20230416+cu117 torchaudio 2.1.0.dev20230416+cu117 torchvision 0.16.0.dev20230416+cu117 tqdm 4.65.0 typing_extensions 4.4.0 tzdata 2023.3 urllib3 1.26.13 Werkzeug 2.2.3 wheel 0.38.4 yapf 0.33.0

[xinyuliu-jeffrey]

mmcv seems not compatible with torch>2.0 (as shown in here.) Please try to downgrade torch first (e.g., we used torch==1.11.0, or follow our installation in the classification here), then install mmcv 1.7 or earlier versions.

[shiyuan7]

try to replace File "/home/xc/anaconda3/envs/seg/lib/python3.10/site-packages/mmcv/parallel/distributed.py", line 160, in _run_ddp_forward module_to_run = self._replicated_tensor_module if \ self._use_replicated_tensor_module else self.module with module_to_run = self.module