search

hellochick / ICNet-tensorflow

TensorFlow-based implementation of "ICNet for Real-Time Semantic Segmentation on High-Resolution Images".
405 stars 153 forks source link

training custom dataset with pre-trained model always get loss NaN...... #89

Open gloomyfish1998 opened 5 years ago

gloomyfish1998 commented 5 years ago

two classes(background and object) training image size = 512 x 256, label image 0 - background, 2- object changes in config.py

  1. number of classes : 2

run with parameters: --update-mean-var --random-scale --random-mirror --dataset cityscapes --filter-scale 1 train output always get whatever change batch_size step 0 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (19.749 sec/step) step 1 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.420 sec/step) step 2 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.453 sec/step) step 3 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.455 sec/step) step 4 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.456 sec/step) step 5 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.646 sec/step) step 6 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.464 sec/step) step 7 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.455 sec/step) step 8 total loss = nan, sub4 = nan, sub24 = nan, sub124 = nan, val_loss: nan (0.458 sec/step)