Error encountered when trying to train Voxel-RCNN using Uni-3D

[Bomsw]

I tried to train Voxel-RCNN using Uni-3D as instructed in readme files, but encountered the following error:

Exception|implicit_gemm]feat=torch.Size([531550, 32]),w=torch.Size([32, 3, 3, 3, 32]),pair=torch.Size([27, 784744]),act=784744,issubm=True,istrain=True
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
epochs:   0%|                                            | 0/30 [00:08<?, ?it/s]
Traceback (most recent call last):
  File "/home/lipw/3DTrans-master/tools/train_multi_db.py", line 261, in <module>
    main()
  File "/home/lipw/3DTrans-master/tools/train_multi_db.py", line 210, in main
    train_func(
  File "/home/lipw/3DTrans-master/tools/train_utils/train_multi_db_utils.py", line 174, in train_model
    accumulated_iter = train_one_epoch(
  File "/home/lipw/3DTrans-master/tools/train_utils/train_multi_db_utils.py", line 59, in train_one_epoch
    loss, tb_dict, disp_dict = model_func(model, batch)
  File "/home/lipw/3DTrans-master/tools/../pcdet/models/__init__.py", line 63, in model_func
    ret_dict, tb_dict, disp_dict = model(batch_dict, **forward_args)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lipw/3DTrans-master/tools/../pcdet/models/detectors/voxel_rcnn.py", line 61, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lipw/3DTrans-master/tools/../pcdet/models/backbones_3d/spconv_backbone_unibn.py", line 211, in forward
    t_conv2_2 = self.conv2_2(t_conv2_1)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/modules.py", line 138, in forward
    input = module(input)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/modules.py", line 138, in forward
    input = module(input)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/conv.py", line 755, in forward
    return self._conv_forward(self.training,
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/conv.py", line 456, in _conv_forward
    out_features = Fsp.implicit_gemm(
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/functional.py", line 224, in forward
    raise e
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/functional.py", line 210, in forward
    out, mask_out, mask_width = ops.implicit_gemm(
  File "/home/lipw/miniconda3/envs/3dtrans/lib/python3.10/site-packages/spconv/pytorch/ops.py", line 1513, in implicit_gemm
    mask_width, tune_res_cpp = ConvGemmOps.implicit_gemm(
RuntimeError: /io/build/temp.linux-x86_64-cpython-310/spconv/build/core_cc/src/cumm/conv/main/ConvMainUnitTest/ConvMainUnitTest_matmul_split_Simt_f32f32f32_0.cu:1047
cuda execution failed with error 700 an illegal memory access was encountered
Simt_f32f32f32f32f32tnt_m32n128k16m32n32k8A1_200_C301LLL_SK error with params [531550, 32] [32, 27, 32] [784744, 32] [27, 784744] [784744, 1] [784744] [] -1

Training PV-RCNN+ using Uni3D or training vanilla Voxel R-CNN works fine though.

[BOBrown]

@Bomsw It seems to be an error in spconv of the BACKBONE_3D. You are using the VoxelBackBone8x or VoxelResBackBone8x?

[Bomsw]

@Bomsw It seems to be an error in spconv of the BACKBONE_3D. You are using the VoxelBackBone8x or VoxelResBackBone8x?

I was using the cfgs/MDF/waymo_nusc/waymo_nusc_voxel_rcnn_feat_3_uni3d.yaml profile as is. It was VoxelBackBone8x_UniBN

[BOBrown]

@Bomsw Have you ever made changes to the code in VoxelBackBone8x_UniBN? You were running using multiple machines or just one machine with multi-gpu?

[Bomsw]

@Bomsw Have you ever made changes to the code in VoxelBackBone8x_UniBN? You were running using multiple machines or just one machine with multi-gpu?

I did not make any changes. I was using one machine with 8 3090s. The issue persisted even with only one gpu (when launching train_multi_db.py directly.

[Bomsw]

@BOBrown So the code ran after I downgraded the spconv version to 2.1.x (was on 2.3.6, spconv 2.2.x did not work either), but it would be quite interesting to know why.

[BOBrown]

@Bomsw My spconv version is 2.1.25

[Bomsw]

@BOBrown OK, so downgrading spconv version works. Thanks a lot!