search

dvlab-research / UVTR

Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)
227 stars 18 forks source link

Prediction values becomes 'Nan' after 1 or 2 epochs #31

Closed javi-99 closed 1 year ago

javi-99 commented 1 year ago

Hi, while training, I receive this error after 1 epoch or sometimes 2 epochs, can you give me a solution for what to do ? thanks in advance

bbox_pred:(tensor(nan, device='cuda:0', grad_fn=), tensor(nan, device='cuda:0', grad_fn=)), cls_score:(tensor(nan, device='cuda:0', grad_fn=), tensor(nan, device='cuda:0', grad_fn=)), gt_bboxes:(tensor(50.7982, device='cuda:0'), tensor(-51.4520, device='cuda:0')), gt_labels:tensor([8, 8, 8, 8, 8, 3, 8, 0, 8, 8, 0, 0, 0, 6, 8, 0, 8, 8, 1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 7, 9, 9], device='cuda:0'), gt_bboxes_ignore:None

Probably the error is caused from here,

mmdet3d_plugin/models/dense_heads/uvtr_head.py", line 265, in _get_target_single sampling_result = self.sampler.sample(assign_result, bbox_pred, UnboundLocalError: local variable 'assign_result' referenced before assignment

javi-99 commented 1 year ago

Decreasing the learning rate resolved this problem.