PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.38k stars 2.84k forks source link

ppyoloe 训练损失函数为nan #7061

Open sensenets123456 opened 1 year ago

sensenets123456 commented 1 year ago

问题确认 Search before asking

Bug组件 Bug Component

Training

Bug描述 Describe the Bug

ppdet\modeling\heads\ppyoloe_head.py def _bbox_loss(self, pred_dist, pred_bboxes, anchor_points, assigned_labels,assigned_bboxes, assigned_scores, assigned_scores_sum): 函数中的代码loss_cls /= assigned_scores_sum 当assigned_scores_sum为0时候,得到loss_cls 的值为inf

复现环境 Environment

linux

Bug描述确认 Bug description confirmation

是否愿意提交PR? Are you willing to submit a PR?

nemonameless commented 1 year ago

请再详细描述下Bug复现步骤、代码改动说明、以及环境信息,谢谢。尤其是环境,几卡总bs的改动。 assigned_scores_sum几乎不大可能为0。如果还不行可以手动设置为1。

sensenets123456 commented 1 year ago

环境:ubantu,从官网拉下的paddle镜像:paddledetection-cuda11.2-cudnn8:v1.0, 2卡,batch size设置为1,图片大小为1920*1920,在citr数据集上训练,