SoheilKoohi commented 4 years ago

5764 # System information

What is the top-level directory of the model you are using: None
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): pip install tensorflow==1.15
TensorFlow version (use command below):1.15 cpu
Bazel version (if compiling from source):None
CUDA/cuDNN version:None
GPU model and memory:None
Exact command to reproduce:

Describe the problem

I am trying to train a SSD_mobilenet object detection model with Tensorflow object detection API. I have cloned the latest version and convert my dataset to tfrecord as the sample provided. Also I used sample config file and just changed the num_classes, fine_tune_checkpoint and path to train and test record. after I run model_main.py after a few iterations the program stops with this error: Traceback (most recent call last):

File "/home/soheilkoohi/models/research/object_detection/model_main.py", line 111, in tf.app.run() File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/absl/app.py", line 299, in run _run_main(main, args) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "/home/soheilkoohi/models/research/object_detection/model_main.py", line 107, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimatorspec , loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(original_exc_info) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/six.py", line 703, in reraise raise value File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(args, kwargs) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate self._evaluate_build_graph(input_fn, hooks, checkpoint_path)) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph self._call_model_fn_eval(input_fn, self.config)) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1544, in _call_model_fn_eval input_fn, ModeKeys.EVAL) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "/home/soheilkoohi/models/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1116, in _call_input_fn return input_fn(kwargs) File "./research/object_detection/inputs.py", line 625, in _eval_input_fn params=params) File "./research/object_detection/inputs.py", line 681, in eval_input raise TypeError('The eval_input_config must be a ' TypeError: The eval_input_config must be a input_reader_pb2.InputReader.

northeastsquare commented 4 years ago

Have you solve it?

northeastsquare commented 4 years ago

OK, get it. Because proto's python files, if generate with different path,

  if not isinstance(train_config, train_pb2.TrainConfig):
    raise TypeError('For training mode, the `train_config` must be a '
                    'train_pb2.TrainConfig.')
  if not isinstance(train_input_config, input_reader_pb2.InputReader):
    raise TypeError('The `train_input_config` must be a '
                    'input_reader_pb2.InputReader.')
  if not isinstance(model_config, model_pb2.DetectionModel):
    raise TypeError('The `model_config` must be a '
                    'model_pb2.DetectionModel.')

will be wrong. So make sure: project directory right, same: research/ pycocotools, object_detection, slim and: protoc -I=. --python_out=. ./*.proto

guaguablue commented 4 years ago

I have met the similar issue, which likes this: python3 model_main.py --model_dir=model/test_path --pipeline_config_path=config/ssd_mobilenet_v1_pets_1.config --num_train_steps=10000 --sample_1_of_n_eval_examples=1 --alsologtostderr

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:Forced number of epochs for all eval validations to be 1. WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered eval_on_train_input_config.num_epochs" = 0. Overwriting "num_epochs to 1. WARNING:tensorflow:Estimator's model_fn (<function create_model_fn..model_fn at 0x7f5a5cec1d90>) includes params argument, but params are not passed to Estimator. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow-1.13.2-py3.6-linux-x86_64.egg/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. =========>train_input_config: label_map_path: "/home/zxg/ssd_training/training_file/label_map/pascal_label_map.pbtxt" tf_record_input_reader { input_path: "/home/zxg/ssd_training/training_file/tfrecords/voc2012_train.record" } <=======>type: <class 'object_detection.protos.input_reader_pb2.InputReader'> <=========

=========>type of train_pb2.TrainConfig: <class 'google.protobuf.pyext.cpp_message.GeneratedProtocolMessageType'> <========= Traceback (most recent call last): File "model_main.py", line 109, in tf.app.run() File "/usr/local/lib/python3.6/dist-packages/tensorflow-1.13.2-py3.6-linux-x86_64.egg/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "model_main.py", line 105, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/training.py", line 471, in train_and_evaluate return executor.run() File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/training.py", line 611, in run return self.run_local() File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/training.py", line 712, in run_local saving_listeners=saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 358, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1151, in _train_model_default input_fn, model_fn_lib.ModeKeys.TRAIN)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 992, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator-1.13.0-py3.6.egg/tensorflow_estimator/python/estimator/estimator.py", line 1079, in _call_input_fn return input_fn(**kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow-1.13.2-py3.6-linux-x86_64.egg/tensorflow/models/research/object_detection/inputs.py", line 452, in _train_input_fn raise TypeError('The "train_input_config" must be a TypeError: The "train_input_config" must be a input_reader_pb2.InputReader.

I find the "models/research/object_detection/inputs.py" file and print the type of "train_input_config“ and ”input_reader_pb2.InputReader" by "print('=========>train_input_config:', train_input_config,'<=======>type:', type(train_input_config), '<=========\n') print('=========>type of train_pb2.TrainConfig:', type(train_pb2.TrainConfig), '<=========')", and I find that they are not the same class type. It seems like the 'voc2012_train.record' file is not suitable for this training. However, when I use “models/research/object_detection/legacy/train.py” to train the model, it works well. So i am confused about the issue. Can you or someone help me please? @northeastsquare

System info: Ubuntu:18.04 Tensorflow:1.13.2 Models:1.13.0

tensorflow / models

"The `eval_input_config` must be a input_reader_pb2.InputReader" in object detection training #8106

5764 # System information

Describe the problem