NAN loss with your network on your data

jhoffman / cycada_release

Code to accompany ICML 2018 paper

BSD 2-Clause "Simplified" License

561 stars 126 forks source link

NAN loss with your network on your data #24

Closed fabvio closed 5 years ago

fabvio commented 5 years ago

Hi, I'm trying to reproduce your results on pixel-level adaptation. So, I'm trying to train the FCN-8s model on the dataset you shared (cyclegta5), but after around 100 iterations, the loss goes nan. This is happening for a 768x768 crop and batch size 2. What can I do to reproduce your results using FCN-8s? Thanks in advance

fabvio commented 5 years ago

Solved setting the right value of the momentum (0.9). For anyone else having my same problem, please note that in the train_fcn.sh file the parameters should be modified following the paper.