matrix contains invalid numeric entries

[zqyJason]

When I train on my custom data, I have met this: File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/base.py", line 233, in train_step losses = self(**data) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func return old_func(*args, **kwargs) File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/base.py", line 167, in forward return self.forward_train(img, img_metas, **kwargs) File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/sparse_rcnn.py", line 63, in forward_train imgs_whwh=imgs_whwh) File "/home/zqy/compare/AdaMixer/mmdet/models/roi_heads/adamixer_decoder.py", line 125, in forward_train gt_labels[i], img_metas[i]) File "/home/zqy/compare/AdaMixer/mmdet/core/bbox/assigners/hungarian_assigner.py", line 132, in assign matched_row_inds, matched_col_inds = linear_sum_assignment(cost) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/scipy/optimize/_lsap.py", line 104, in linear_sum_assignment return _lsap_module.calculate_assignment(cost_matrix) ValueError: matrix contains invalid numeric entries It seems that the loss is inf, I have no idea ,can anyone help?

[sebgao]

Can you please check whether there is at least one bounding box in every image across the training dataset? It seem there is an image that contains no objects in the annotation.

[zqyJason]

Can you please check whether there is at least one bounding box in every image across the training dataset? It seem there is an image that contains no objects in the annotation.

Thank you! I found the training set does have empty annotations.