search

scaelles / OSVOS-TensorFlow

One-Shot Video Object Segmentation
http://vision.ee.ethz.ch/~cvlsegmentation/osvos/
GNU General Public License v3.0
438 stars 132 forks source link

trainning loss=NAN #24

Closed Adarine99 closed 6 years ago

Adarine99 commented 6 years ago

Hi, I'm sorry to disturb you again，but I have no clue how to solve this.Testing on my own data,I use one human segmentation image and origin images as input, then run the demo.py , got another problem as follows:

Started loading Performing the data augmentation Done initializing Dataset Initializing from specified pre-trained model... INFO:tensorflow:Restoring parameters from models/OSVOS_parent/OSVOS_parent.ckpt-50000 Weights initialized Start training 2018-09-01 15:43:31.268901 Iter 10: Training Loss = 232345665536.0000 2018-09-01 15:45:39.789833 Iter 20: Training Loss = nan 2018-09-01 15:47:21.016101 Iter 30: Training Loss = nan 2018-09-01 15:49:22.912398 Iter 40: Training Loss = nan 2018-09-01 15:51:17.401571 Iter 50: Training Loss = nan Finally,Igot the output images,and the result is terrible. Is there something wrong?Is there anyone can help me? Thank you so much!

kmaninis commented 6 years ago

Hi, it could be one of the following things:

  1. Learning rate is too high and should be reduced (default is 1e-8, please try 1e-9 or 1e-10).
  2. Your label is not binary in case you don't use the label preprocessing of this code. Please make sure your label only has values of {0, 1}.

I hope this helps.

Adarine99 commented 6 years ago

@kmaninis Thank you for your reply,I believe my problem is No.2. Really save me:)

thiagoluange commented 5 years ago

Thank you , my problem is No. 1( worked with 1e-9). :)