[Bug] Running Oriented RepPoints happens with RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

[leily578]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

Python: 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)] CUDA available: True GPU 0: NVIDIA GeForce RTX 2080 Ti CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8 NVCC: Cuda compilation tools, release 11.8, V11.8.89 MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.37.32824 版 GCC: n/a PyTorch: 2.0.1+cu118 PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 193431937
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX2
• CUDA Runtime 11.8
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.7
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.15.2+cu118 OpenCV: 4.8.0 MMCV: 1.7.1 MMCV Compiler: MSVC 193732824 MMCV CUDA Compiler: 11.8

Reproduces the problem - code sample

./tools/train.py configs/oriented_reppoints/oriented_reppoints_r50_fpn_1x_dota_le135.py

Reproduces the problem - command or script

python ./tools/train.py configs/oriented_reppoints/oriented_reppoints_r50_fpn_1x_dota_le135.py

Reproduces the problem - error message

C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. warnings.warn( C:\Anaconda3\envs\mmrotate\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:3484.) return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] Traceback (most recent call last): File "C:\Code\mmrotate\tools\train.py", line 194, in main() File "C:\Code\mmrotate\tools\train.py", line 183, in main train_detector( File "c:\code\mmrotate\mmrotate\apis\train.py", line 144, in train_detector runner.run(data_loaders, cfg.workflow) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 136, in run epoch_runner(data_loaders[i], kwargs) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 53, in train self.run_iter(data_batch, train_mode=True, kwargs) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 31, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv\parallel\data_parallel.py", line 77, in train_step return self.module.train_step(inputs[0], kwargs[0]) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmdet\models\detectors\base.py", line 248, in train_step losses = self(data) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmcv\runner\fp16_utils.py", line 119, in new_func return old_func(args, kwargs) File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmdet\models\detectors\base.py", line 172, in forward return self.forward_train(img, img_metas, kwargs) File "c:\code\mmrotate\mmrotate\models\detectors\single_stage.py", line 81, in forward_train losses = self.bbox_head.forward_train(x, img_metas, gt_bboxes, File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmdet\models\dense_heads\base_dense_head.py", line 335, in forward_train losses = self.loss(loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore) File "c:\code\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 874, in loss cls_reg_targets_init = self.get_targets( File "c:\code\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 798, in get_targets sampling_result) = multi_apply( File "C:\Anaconda3\envs\mmrotate\lib\site-packages\mmdet\core\utils\misc.py", line 30, in multi_apply return tuple(map(list, zip(*map_results))) File "c:\code\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 675, in _point_target_single assign_result = assigner.assign(proposals, gt_bboxes, overlaps, File "c:\code\mmrotate\mmrotate\core\bbox\assigners\convex_assigner.py", line 125, in assign points_index = points_range[lvl_idx] RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Additional information

It could be that something is wrong with a parameter, but I don't konw which one.

[silencersai]

same issue

[bayuezha]

This bug has been fixed in main branch but not in the release of 34 (installed by 'pip install mmrotate'), you can remove it then download the source code of mmrotate and update it as follows.

git clone https://github.com/open-mmlab/mmrotate.git cd mmrotate pip install -r requirements/build.txt pip install -v -e .