search

aurora95 / Keras-FCN

Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation（Unfinished）
MIT License
650 stars 268 forks source link

SegDataGenerator validation w/ callbacks #9

Open ahundt opened 7 years ago

ahundt commented 7 years ago

I tried enabling the validation code in SegDataGenerator plus using the keras early stopping and ReduceLROnPlateau which I put in train.py, uncommenting the validation lines in my validation_callbacks branch forked from your repo.

Unfortunately it seems to fail with an error like the following:

  File "train.py", line 132, in <module>                                                                                                [1170/7924]
    data_dir, label_dir, target_size=target_size, batchnorm_momentum=batchnorm_momentum, resume_training=resume_training)
  File "train.py", line 106, in train
    nb_val_samples = 64
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.2-py2.7.egg/keras/legacy/interfaces.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.2-py2.7.egg/keras/engine/training.py", line 1913, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.2-py2.7.egg/keras/callbacks.py", line 75, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/usr/local/lib/python2.7/dist-packages/Keras-2.0.2-py2.7.egg/keras/callbacks.py", line 477, in on_epoch_end
    if self.monitor_op(current - self.min_delta, self.best):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

Any ideas why the monitor_op might be NoneType and not a value that can be used to correctly handle the callbacks?

Perhaps it is related to the custom sparse_accuracy_ignoring_last_label and the corresponding loss function... I started debugging but it looks like the monitor_op is correctly assigned.