Inf loss detected for Loc

biubug6 / Pytorch_Retinaface

Retinaface get 80.99% in widerface hard val using mobilenet0.25.

MIT License

2.63k stars 774 forks source link

Inf loss detected for Loc #10

Open xsacha opened 5 years ago

xsacha commented 5 years ago

While training, I hit an Inf loss for the bounding boxes.

Epoch:162/300 || Epochiter: 57/101 || Iter: 16318/30300 || Loc: inf Cla: 1.8687 Landm: 4.1464 || LR: 0.00100000 || Batchtime: 5.3929 s || ETA: 20:56:48

rydenisbak commented 5 years ago

I think after augmentation you got some gt bbox with size equal zero

xsacha commented 5 years ago

Should this be filtered out to avoid it affecting result?

biubug6 commented 5 years ago

Sorry, I can't reproduce the problem. Gt bbox with size equal zero has been filtered out in line 66 of "data_augment.py".

twmht commented 5 years ago

@xsacha

I have same issue. Did you solve?

xsacha commented 5 years ago

Sorry, I didn't solve it

hegc commented 4 years ago

when loc loss is inf at one iter, I don't cal & update grad.