Nan for valid acc - Githubissues

meijieru commented 6 years ago

When training on pascal 3d+ following the readme, the training acc is normal while the valid acc is nan.

==> Creating model from file: models/hg-stacked.lua                                                                                                                                                              
==> Converting model to CUDA                                                                                                                                                                                     
==> Starting epoch: 1/100                                                                                                                                                                                        
 [==>..................................... 174/4000 ....................................]  ETA: 9m20s | Step: 146ms                                                                                              
 [======================================== 4000/4000 ==================================>]  Tot: 10m30s | Step: 151ms                                                                                             
      train : Loss: 0.0005612 Acc: 0.0629                                                                                                                                                                        
 [======================================== 2958/2958 ==================================>]  Tot: 2m48s | Step: 54ms                                                                                               
      valid : Loss: 0.0002669 Acc: nan                                                                                                                                                                           
==> Starting epoch: 2/100                                                                                                                                                                                        
 [======================================== 4000/4000 ==================================>]  Tot: 10m38s | Step: 158ms                                                                                             
      train : Loss: 0.0004416 Acc: 0.1401                                                                                                                                                                        
 [======================================== 2958/2958 ==================================>]  Tot: 2m46s | Step: 56ms                                                                                               
      valid : Loss: 0.0002426 Acc: nan                                                                                                                                                                           
==> Starting epoch: 3/100                                                                                                                                                                                        
 [======================================== 4000/4000 ==================================>]  Tot: 10m42s | Step: 164ms                                                                                             
      train : Loss: 0.0004159 Acc: 0.2499                                                                                                                                                                        
 [======================================== 2958/2958 ==================================>]  Tot: 2m47s | Step: 57ms                                                                                               
      valid : Loss: 0.0023763 Acc: nan

xiaojieli0903 commented 5 years ago

Hello, I met the same problem. Have you solved this issue?

meijieru commented 5 years ago

Not yet.

maximpavliv commented 4 years ago

Same for me, any news on this?

geopavlakos / object3d

Nan for valid acc #4