@Vegeta2020
while trying to train the model I am getting an error message "CUDA error: an illegal memory access was encountered"
and the training is flailing.
File "/mnt/appl/software/PyTorch/1.7.1-fosscuda-2020b/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/gebreawe/Code/SE-SSD/det3d/models/detectors/voxelnet_sessd.py", line 41, in forward
return self.bbox_head.loss(example, preds, is_ema[1])
File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 712, in loss
consistency_loss = self.consistency_loss(preds_dicts, preds_ema, example)
File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 682, in consistency_loss
box_consistency_loss, idx1, idx2, mask1, mask2 = self.nn_distance(top_box_preds_stu, top_box_preds_tea)
File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 586, in nn_distance
ans_iou = ans_iou[mask1]
RuntimeError: CUDA error: an illegal memory access was encountered
@Vegeta2020 while trying to train the model I am getting an error message "CUDA error: an illegal memory access was encountered" and the training is flailing.
I am using: Pytorch: 1.7.1 Tesla V 100 GPUs
Has anyone faced a similar problem?
File "/mnt/appl/software/PyTorch/1.7.1-fosscuda-2020b/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/gebreawe/Code/SE-SSD/det3d/models/detectors/voxelnet_sessd.py", line 41, in forward return self.bbox_head.loss(example, preds, is_ema[1]) File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 712, in loss consistency_loss = self.consistency_loss(preds_dicts, preds_ema, example) File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 682, in consistency_loss box_consistency_loss, idx1, idx2, mask1, mask2 = self.nn_distance(top_box_preds_stu, top_box_preds_tea) File "/home/gebreawe/Code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 586, in nn_distance ans_iou = ans_iou[mask1] RuntimeError: CUDA error: an illegal memory access was encountered