[X] I have searched related issues but cannot get the expected help.
[X] 2. I have read the FAQ documentation but cannot get the expected help.
[X] 3. The bug has not been fixed in the latest version.
Describe the bug
Issue is RuntimeError: get_indice_pairs is not implemented on CPU
root@aa4559b96fbd:~/workspace# python3 mmdeploy/tools/deploy.py "mmdeploy/configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-nus-64x4.py" "centerpoint_voxel01_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py" "centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_030004-9061688e.pth" "samples/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin" --work-dir mmdeploy_model/centerpoint --device "cuda:0"
04/04 20:02:08 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
/usr/local/lib/python3.8/dist-packages/mmdet3d/evaluation/functional/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
04/04 20:02:11 - mmengine - WARNING - Failed to search registry with scope "mmdet3d" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized.
04/04 20:02:11 - mmengine - WARNING - Failed to search registry with scope "mmdet3d" in the "mmdet3d_tasks" registry tree. As a workaround, the current "mmdet3d_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_030004-9061688e.pth
04/04 20:02:12 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
device cuda:0
04/04 20:02:12 - mmengine - INFO - Export PyTorch model to ONNX: mmdeploy_model/centerpoint/end2end.onnx.
04/04 20:02:12 - mmengine - WARNING - Can not find torch.nn.functional._scaled_dot_product_attention, function rewrite will not be applied
/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_ops.py:91: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if ndim == 2:
/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_ops.py:93: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
elif ndim == 3:
============= Diagnostic Run torch.onnx.export version 2.0.0+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
ret = func(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 99, in torch2onnx
export(
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
return self.call_function(func_name_, *args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
return self.call_function_local(func_name, *args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
return pipe_caller(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
ret = func(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/export.py", line 138, in export
torch.onnx.export(
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1548, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/optimizer.py", line 27, in model_to_graph__custom_optimizer
graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1113, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 989, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 893, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 1268, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/export.py", line 123, in wrapper
return forward(*arg, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 83, in mvxtwostagedetector__forward
_, pts_feats = self.extract_feat(batch_inputs_dict=batch_inputs_dict)
File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 42, in mvxtwostagedetector__extract_feat
pts_feats = self.extract_pts_feat(
File "/usr/local/lib/python3.8/dist-packages/mmdet3d/models/detectors/mvx_two_stage.py", line 214, in extract_pts_feat
x = self.pts_middle_encoder(voxel_features, voxel_dict['coors'],
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.8/dist-packages/mmdet3d/models/middle_encoders/sparse_encoder.py", line 145, in forward
x = self.conv_input(input_sp_tensor)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_modules.py", line 135, in forward
input = module(input)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_conv.py", line 157, in forward
outids, indice_pairs, indice_pair_num = ops.get_indice_pairs(
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_ops.py", line 99, in get_indice_pairs
return get_indice_pairs_func(indices, batch_size, out_shape,
RuntimeError: get_indice_pairs is not implemented on CPU
04/04 20:02:14 - mmengine - ERROR - /root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.pytorch2onnx.torch2onnx` with Call id: 0 failed. exit.
root@aa4559b96fbd:~/workspace# python3 mmdeploy/tools/check_env.py
04/04 20:09:09 - mmengine - INFO -
04/04 20:09:09 - mmengine - INFO - **********Environmental information**********
04/04 20:09:10 - mmengine - INFO - sys.platform: linux
04/04 20:09:10 - mmengine - INFO - Python: 3.8.10 (default, May 26 2023, 14:05:08) [GCC 9.4.0]
04/04 20:09:10 - mmengine - INFO - CUDA available: True
04/04 20:09:10 - mmengine - INFO - MUSA available: False
04/04 20:09:10 - mmengine - INFO - numpy_random_seed: 2147483648
04/04 20:09:10 - mmengine - INFO - GPU 0: NVIDIA RTX A6000
04/04 20:09:10 - mmengine - INFO - CUDA_HOME: /usr/local/cuda
04/04 20:09:10 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.8, V11.8.89
04/04 20:09:10 - mmengine - INFO - GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
04/04 20:09:10 - mmengine - INFO - PyTorch: 2.0.0+cu118
04/04 20:09:10 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.7
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
04/04 20:09:10 - mmengine - INFO - TorchVision: 0.15.0+cu118
04/04 20:09:10 - mmengine - INFO - OpenCV: 4.5.4
04/04 20:09:10 - mmengine - INFO - MMEngine: 0.10.3
04/04 20:09:10 - mmengine - INFO - MMCV: 2.1.0
04/04 20:09:10 - mmengine - INFO - MMCV Compiler: GCC 9.3
04/04 20:09:10 - mmengine - INFO - MMCV CUDA Compiler: 11.8
04/04 20:09:10 - mmengine - INFO - MMDeploy: 1.3.1+
04/04 20:09:10 - mmengine - INFO -
04/04 20:09:10 - mmengine - INFO - **********Backend information**********
04/04 20:09:10 - mmengine - INFO - tensorrt: 8.6.1
04/04 20:09:10 - mmengine - INFO - tensorrt custom ops: Available
04/04 20:09:10 - mmengine - INFO - ONNXRuntime: None
04/04 20:09:10 - mmengine - INFO - ONNXRuntime-gpu: 1.15.1
04/04 20:09:10 - mmengine - INFO - ONNXRuntime custom ops: Available
04/04 20:09:10 - mmengine - INFO - pplnn: 0.8.1
04/04 20:09:10 - mmengine - INFO - ncnn: 1.0.20230905
04/04 20:09:10 - mmengine - INFO - ncnn custom ops: Available
04/04 20:09:10 - mmengine - INFO - snpe: None
04/04 20:09:10 - mmengine - INFO - openvino: 2023.0.2
04/04 20:09:10 - mmengine - INFO - torchscript: 2.0.0+cu118
04/04 20:09:10 - mmengine - INFO - torchscript custom ops: Available
04/04 20:09:10 - mmengine - INFO - rknn-toolkit: None
04/04 20:09:10 - mmengine - INFO - rknn-toolkit2: None
04/04 20:09:10 - mmengine - INFO - ascend: None
scikit-learn version 1.3.2 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
04/04 20:09:11 - mmengine - INFO - coreml: 7.1
04/04 20:09:11 - mmengine - INFO - tvm: None
04/04 20:09:11 - mmengine - INFO - vacc: None
04/04 20:09:11 - mmengine - INFO -
04/04 20:09:11 - mmengine - INFO - **********Codebase information**********
04/04 20:09:11 - mmengine - INFO - mmdet: 3.3.0
04/04 20:09:11 - mmengine - INFO - mmseg: None
04/04 20:09:11 - mmengine - INFO - mmpretrain: 1.2.0
04/04 20:09:11 - mmengine - INFO - mmocr: None
04/04 20:09:11 - mmengine - INFO - mmagic: None
04/04 20:09:11 - mmengine - INFO - mmdet3d: 1.4.0
04/04 20:09:11 - mmengine - INFO - mmpose: None
04/04 20:09:11 - mmengine - INFO - mmrotate: None
04/04 20:09:11 - mmengine - INFO - mmaction: None
04/04 20:09:11 - mmengine - INFO - mmrazor: None
04/04 20:09:11 - mmengine - INFO - mmyolo: None
### Error traceback
```Shell
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
ret = func(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 99, in torch2onnx
export(
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
return self.call_function(func_name_, *args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
return self.call_function_local(func_name, *args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
return pipe_caller(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
ret = func(*args, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/export.py", line 138, in export
torch.onnx.export(
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1548, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/optimizer.py", line 27, in model_to_graph__custom_optimizer
graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 1113, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 989, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 893, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 1268, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/apis/onnx/export.py", line 123, in wrapper
return forward(*arg, **kwargs)
File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 83, in mvxtwostagedetector__forward
_, pts_feats = self.extract_feat(batch_inputs_dict=batch_inputs_dict)
File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 42, in mvxtwostagedetector__extract_feat
pts_feats = self.extract_pts_feat(
File "/usr/local/lib/python3.8/dist-packages/mmdet3d/models/detectors/mvx_two_stage.py", line 214, in extract_pts_feat
x = self.pts_middle_encoder(voxel_features, voxel_dict['coors'],
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.8/dist-packages/mmdet3d/models/middle_encoders/sparse_encoder.py", line 145, in forward
x = self.conv_input(input_sp_tensor)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_modules.py", line 135, in forward
input = module(input)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_conv.py", line 157, in forward
outids, indice_pairs, indice_pair_num = ops.get_indice_pairs(
File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/sparse_ops.py", line 99, in get_indice_pairs
return get_indice_pairs_func(indices, batch_size, out_shape,
RuntimeError: get_indice_pairs is not implemented on CPU
Checklist
Describe the bug
Issue is
RuntimeError: get_indice_pairs is not implemented on CPU
Reproduction
Environment