search

stefan-ainetter / grasp_det_seg_cnn

Code for ICRA21 paper "End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGB".
BSD 3-Clause "New" or "Revised" License
133 stars 17 forks source link

RuntimeError: CUDA error: device-side assert triggered #31

Open saad-abdullah62 opened 12 months ago

saad-abdullah62 commented 12 months ago

Hi , I am training this network on a dataset with three classes only, therefore i change num_semantic from 31 to 4. But I get the following error.

File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 684, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/RBC-2285-new_conveyor_ai/grasp_det_seg_cnn/grasp_det_seg/models/det_seg.py", line 100, in forward
    self.semantic_seg_algo.training(self.sem_head, x, sem, valid_size, img_size)
  File "/RBC-2285-new_conveyor_ai/grasp_det_seg_cnn/grasp_det_seg/algos/semantic_seg.py", line 94, in training
    conf_mat = self._confusion_matrix(sem_pred, sem)
  File "/RBC-2285-new_conveyor_ai/grasp_det_seg_cnn/grasp_det_seg/algos/semantic_seg.py", line 63, in _confusion_matrix
    valid = sem_i != self.ignore_index # get index where gt is present
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 27, in wrapped
    return f(*args, **kwargs)
RuntimeError: CUDA error: device-side assert triggered

Do I need to change semantic classes somewhere else in code? I have a binary mask of image where each of 3 object's intensity corresponds to either 1,2 or 3.

Also is there any way to only train the grasp head and not the semantic segmentation head?