search

wayveai / fiery

PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"
https://wayve.ai/blog/fiery-future-instance-prediction-birds-eye-view
MIT License
549 stars 87 forks source link

NAN loss after one epoch #20

Open sjbling opened 2 years ago

sjbling commented 2 years ago

HI, I train the model based on nuscenes with baseline.yml, but I got the nan loss after one epoch, how to train the model from scratch? Looking forward to your reply！

markus93 commented 2 years ago

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

I also tried the same setup and have the same issue. I tried to dig a little bit deeper and found out that running means and running variances of batch normalization are getting first to -infinity and then to nan. This also causes the loss to go to nan. The loss seems to get back from the nan-values, however, the model will still output -infinities for the segmentation.

Best regards, Markus

anthonyhu commented 2 years ago

Hey both of you, and sorry for the late answer. It is a known issue, and it seems like training the whole network from scratch leads to instability. The fix is to load pre-trained weight from a 1-timestep model (FIERY Static) first because training the whole future prediction model as discussed here: https://github.com/wayveai/fiery/issues/8

cs-hue commented 1 year ago

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

After I train the model for epoch I got loss as negative value.Do you know the reason?

Best regards, Cara

anthonyhu commented 1 year ago

Hey Cara, the negative loss value is due to the adaptive weighting of the losses using uncertainty. There is an additional loss term that prevents the weight from growing too large (see page 5 https://arxiv.org/pdf/1705.07115.pdf)