search

lilanxiao / Rotated_IoU

Differentiable IoU of rotated bounding boxes using Pytorch
MIT License
415 stars 64 forks source link

Function 'SqrtBackward' returned nan values in its 0th output. Bug in min_enclosing_box.py? #20

Closed zye1996 closed 3 years ago

zye1996 commented 3 years ago

When backprop with GIoU loss, there is a sqrt out of range regarding the line here when sqrt encounter 0 values.

Should we add a small offset to the value inside the sqrt? I tried 1e-8 and the training became unstable while 1e-16 is fine.

num = torch.sqrt( (y2-y1).square() + (x2-x1).square() +1e-16) + 1e-8

lilanxiao commented 3 years ago

Thank you very much for the issue!

Yes, it's a bug. According to this link: https://github.com/pytorch/pytorch/issues/6394, the backprop of torch.sqrt() would generate nan if the input is zero.

Actually, I'm surprised that Pytorch really puts an inf there. I thought the gradient was hard-coded to some very large but limited value. lol. My computer is now occupied by some other tasks and I cannot run tests. But I will fix this ASAP and let you know.

lilanxiao commented 3 years ago

hi, I've changed that line to num = torch.sqrt( (y2-y1).square() + (x2-x1).square() +1e-14) the 1e-8 is no more necessary as the sqrt is guaranteed positive. Please let me know if there are further issues.

zye1996 commented 3 years ago

hi, I've changed that line to num = torch.sqrt( (y2-y1).square() + (x2-x1).square() +1e-14) the 1e-8 is no more necessary as the sqrt is guaranteed positive. Please let me know if there are further issues.

Hi I have verified the fix work for the backprop. I can confirm that DIoU loss enhance the performance of the detector considerably.