resume training for object detection

[junweima]

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• What is the top-level directory of the model you are using: object detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): tensorflow-gpu=1.9.0
• Bazel version (if compiling from source): n/a
• CUDA/cuDNN version: 9.0
• GPU model and memory: nvidia 1080ti
• Exact command to reproduce: python model_main.py

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

I'm training ssdlite model with mobileNetV2 on coco dataset, everything is default. The training was interrupted and killed and now I want to restore the model and resume training.

However, the current script does not support this feature. I tried to set fine_tune_checkpoint_type: "detection", the model is loaded but the script only runs evaluation without training. Is there a way to resume training?

I've seen this https://github.com/tensorflow/models/issues/4116 post but I'm not sure which part was modified and train.py seemed to be moved to legacy folder now.

Specifically, I just want to ask 1. Is there a feature for resuming training? If so, which flags should I set? 2. If there is no such feature, could you give some pointers on how to add this feature? I can make a PR if needed.

Thanks.

[junweima]

Hi, I solved it by passing a larger value or set None for NUM_TRAIN_STEPS and NUM_EVAL_STEPS. I didn't know train_steps is set to max_steps in function create_train_and_eval_specs in model_lib.py

train_spec = tf.estimator.TrainSpec( input_fn=train_input_fn, max_steps=train_steps)