search

traveller59 / second.pytorch

SECOND for KITTI/NuScenes object detection
MIT License
1.71k stars 721 forks source link

A problem about using RTX 2080ti to evaluate #398

Open 940023303 opened 4 years ago

940023303 commented 4 years ago

hello, @traveller59. Recently, I have a problem, and I need your help. The code can run in GPU RTX1080ti or GTX TITANx, but an error will occur when using RTX 2080ti. I pulled the code in 2019-08 and made some modifications. I have no idea to deal with this problem. Can you give me some suggestions? Do I need to pull the latest code? Thank you.

  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/autolab/tianxuebo/second.pytorch/second/pytorch/models/voxelnet.py", line 364, in forward
    preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev)
  File "/home/autolab/tianxuebo/second.pytorch/second/pytorch/models/voxelnet.py", line 333, in network_forward
    voxel_features, coors, batch_size)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/autolab/tianxuebo/second.pytorch/second/pytorch/models/middle.py", line 203, in forward
    ret = self.middle_conv(ret)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/spconv/modules.py", line 130, in forward
    input = module(input)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/spconv/conv.py", line 177, in forward
    use_hash=self.use_hash)
  File "/home/autolab/anaconda3/envs/second/lib/python3.7/site-packages/spconv/ops.py", line 93, in get_indice_pairs
    stride, padding, dilation, out_padding, int(subm), int(transpose), int(use_hash))
RuntimeError: /home/autolab/tianxuebo/second.pytorch/second/spconv/src/spconv/indice.cu 165
cuda execution failed with error 8 invalid device function
prepareSubMHashKernel failed
File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tiev-plus/tiev-plus/modules/src/lidar/second.pytorch/second/pytorch/models/voxelnet.py", line 364, in forward
    preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev)
  File "/home/tiev-plus/tiev-plus/modules/src/lidar/second.pytorch/second/pytorch/models/voxelnet.py", line 333, in network_forward
    voxel_features, coors, batch_size)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tiev-plus/tiev-plus/modules/src/lidar/second.pytorch/second/pytorch/models/middle.py", line 203, in forward
    ret = self.middle_conv(ret)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/spconv/modules.py", line 134, in forward
    input.features = module(input.features)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 50, in forward
    return F.threshold(input, self.threshold, self.value, self.inplace)
  File "/home/tiev-plus/.virtualenvs/second/lib/python3.6/site-packages/torch/nn/functional.py", line 840, in threshold
    result = _VF.threshold(input, threshold, value)
RuntimeError: CUDA error: invalid argument
940023303 commented 4 years ago

I fixed this problem. Because of the old Spconv code relying on the torch 1.0, I installed Torch 1.0 on my computer with 2080ti GPU. The second code can not run on the computer with an old version torch. I reinstall the newest torch and modify the Spconv code to pass the build step. after reinstalled the Spconv, this error is disappearance.