Loss is Nan, VGG backbone not working

lzccccc / 3d-bounding-box-estimation-for-autonomous-driving

3d bounding box estimation from monocular image based on 2d bounding box

127 stars 36 forks source link

Loss is Nan, VGG backbone not working #16

Open ghost opened 3 years ago

ghost commented 3 years ago

I was able to setup and run the training, but if I use VGG as backbone, the training log looked like below:

ETA: 3:15:50 - loss: inf - dimensions_loss: inf - orientation_loss: 1.74 ETA: 2:11:50 - loss: nan - dimensions_loss: nan - orientation_loss: 1.69 TA: 1:39:53 - loss: nan - dimensions_loss: nan - orientation_loss: 1.34

MobileNetV2 works fine. But the performance is not as good as VGG. I really need to use VGG.

KenYu910645 commented 2 years ago

I encountered the same problem and solved it by change the optimizer from SGD to Adams. For anyone wants to try this, change the Line 43 in train.py. minimizer = optimizer.Adam(learning_rate=0.0001) The dimension loss will still be very high in the first epoch, but it'll become fairly small after that.