Closed lkydsb21 closed 1 year ago
mmdetection/mmdet/models/task_modules/assigners/hungarian_assigner.py
changed the inf value in the cost to -1e8
I don't think there's a problem with my dataset because I can training on other network models such as faster rcnn and ssd
The potential reason is that the model does not converge.
I melt the same problem when I use dab_detr: configs/dab_detr/dab-detr_r50_8xb2-50e_coco.py.
The error log: File "/home/zhangjw/research/research/mmdetection2023/mmdetection/mmdet/models/dense_heads/detr_head.py", line 406, in _get_targets_single assign_result = self.assigner.assign( File "/home/zhangjw/research/research/mmdetection2023/mmdetection/mmdet/models/task_modules/assigners/hungarian_assigner.py", line 135, in assign matched_row_inds, matched_col_inds = linear_sum_assignment(cost) File "/root/anaconda3/lib/python3.8/site-packages/scipy/optimize/_lsap.py", line 93, in linear_sum_assignment raise ValueError("matrix contains invalid numeric entries") ValueError: matrix contains invalid numeric entries
My dataset can be trained with many other models, such as Mask R-CNN, AutoAssign, and rtmdet.
At last, I found that this may be caused by two issues:
These issues may cause the model not to converge.
I successfully resolved this bug by re-installing mmdetection. It's worth noting that you should not modify the source code easily. I once encountered different errors because I had modified the source code before. Try to find possible issues from your own configuration file and dataset.
When I trained my own dataset with DETR, I got valueerror: matrix contains invalid numeric entries, according to the previous answer, I used a pre-trained model but this problem still occurs, so I changed the inf value in the cost to -1e8, and the model can start training, but the loss value is nan. how can I fix it. 20230526_134111.log