loss going to nan after 60k steps

thtrieu / darkflow

Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices

GNU General Public License v3.0

6.13k stars 2.08k forks source link

loss going to nan after 60k steps #793

Open alvinxiii opened 6 years ago

alvinxiii commented 6 years ago

What's the problem my training goes nan after 60k steps? I found several comments about this problem. For sure, my problem is not related to the xml label issue, or training goes nan after step 1. I train the dataset by using yolo.weights and a custom yolo-1c.cfg files. Is there any solution to solve this problem? Really appreciate if someone can help.

Jped commented 6 years ago

When I change the --trainer to --trainer adam I do not run into this problem....not sure why that makes a difference.