[DeepLab] Add Early Stopping support

fostiropoulos commented 5 years ago

System information

What is the top-level directory of the model you are using: tensorflow/models/research/deeplab
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): b'unknown' 1.13.1
Bazel version (if compiling from source):
CUDA/cuDNN version: 10.1
GPU model and memory: 1080ti

Exact command to reproduce:

python "${WORK_DIR}"/train.py \
--logtostderr \
--train_split="train" \
--model_variant="xception_65" \
--num_clones=2 \
--atrous_rates=6 \
--atrous_rates=12 \
--atrous_rates=18 \
--output_stride=16 \
--decoder_output_stride=4 \
--train_crop_size="513,513" \
--train_batch_size=4 \
--training_number_of_steps="${NUM_ITERATIONS}" \
--fine_tune_batch_norm=true \
--tf_initial_checkpoint="${INIT_FOLDER}/model.ckpt" \
--train_logdir="${TRAIN_LOGDIR}" \
--dataset_dir="${DATASET_DIR}"

Describe the problem

Loss is increasing drastically after a specific number of steps when transfer learning. To make training easier I propose early stopping is implemented and added as an option.

Source code / logs

sumukhaithal6 commented 5 years ago

@aquariusjay Can I work on this?

fostiropoulos commented 5 years ago

@sumukhaithal6 @aquariusjay I already implemented it myself. Do you want me to create a PR? I would like the opinion of everybody first that they are happy with this feature implementation and find it useful. @gowthamkpr

sw5park commented 4 years ago

@fostiropoulos

Hello, could you kindly point me in the direction to the implementation of adding early stopping support ?

Thank you for the consideration

tensorflow / models