Not on the same device error in PointAssigner

[hujh1994]

Use tools/train.py to train RepPoints Network, and you will get RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu) The last lines of Traceback is File "xxxxxxxx\mmdetection\mmdet\models\task_modules\assigners\point_assigner.py", line 114, in assign points_index = points_range[lvl_idx] According to my Github Desktop and memory, I have not modified any code of mmdetection. I use CityPersons Dataset. However, I believe the problem is not relative to the dataset.

Environment sys.platform: win32 Python: 3.10.9 | packaged by Anaconda, Inc. | (main, Mar 1 2023, 18:18:15) [MSC v.1916 64 bit (AMD64)] CUDA available: True numpy_random_seed: 2147483648 GPU 0: NVIDIA GeForce RTX 3050 Laptop GPU CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6 NVCC: Cuda compilation tools, release 11.6, V11.6.124 MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.35.32217.1 版 GCC: n/a PyTorch: 1.13.1 PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192829337
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX2
• CUDA Runtime 11.6
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.3.2 (built against CUDA 11.5)
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.14.1 OpenCV: 4.7.0 MMEngine: 0.7.3 MMDetection: 3.0.0+ecac3a7

Error traceback Traceback (most recent call last): File "path_to_my_project\train.py", line 133, in main() File "path_to_my_project\train.py", line 129, in main runner.train() File "C:\Users\HuShi\anaconda3\lib\site-packages\mmengine\runner\runner.py", line 1721, in train model = self.train_loop.run() # type: ignore File "C:\Users\HuShi\anaconda3\lib\site-packages\mmengine\runner\loops.py", line 278, in run self.run_iter(data_batch) File "C:\Users\HuShi\anaconda3\lib\site-packages\mmengine\runner\loops.py", line 301, in run_iter outputs = self.runner.model.train_step( File "C:\Users\HuShi\anaconda3\lib\site-packages\mmengine\model\base_model\base_model.py", line 114, in train_step losses = self._run_forward(data, mode='loss') # type: ignore File "C:\Users\HuShi\anaconda3\lib\site-packages\mmengine\model\base_model\base_model.py", line 340, in _run_forward results = self(*data, mode=mode) File "C:\Users\HuShi\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, *kwargs) File "xxxxx\mmdetection\mmdet\models\detectors\base.py", line 92, in forward return self.loss(inputs, data_samples) File "xxxxx\mmdetection\mmdet\models\detectors\single_stage.py", line 78, in loss losses = self.bbox_head.loss(x, batch_data_samples) File "xxxxx\mmdetection\mmdet\models\dense_heads\base_dense_head.py", line 123, in loss losses = self.loss_by_feat(loss_inputs) File "xxxxx\mmdetection\mmdet\models\dense_heads\reppoints_head.py", line 705, in loss_by_feat cls_reg_targets_init = self.get_targets( File "xxxxx\mmdetection\mmdet\models\dense_heads\reppoints_head.py", line 561, in get_targets sampling_results_list) = multi_apply( File "xxxxx\mmdetection\mmdet\models\utils\misc.py", line 219, in multi_apply return tuple(map(list, zip(*map_results))) File "xxxxx\warpnet\mmdetection\mmdet\models\dense_heads\reppoints_head.py", line 444, in _get_targets_single
assign_result = assigner.assign(pred_instances, gt_instances, File "xxxxx\warpnet\mmdetection\mmdet\models\task_modules\assigners\point_assigner.py", line 114, in assign
points_index = points_range[lvl_idx] RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Bug fix Initializing points_range with the device of points_lvl will solve the problem. I will create a PR to fix it.

[shankar-vision-eng]

I have run into the same issue, looks like an upgrade to torch caused this issue as issue as not present in torch1.10. This is the check in torch that throws the error. Any way this can be fixed in next release ?