search

tensorflow / models

Models and examples built with TensorFlow
Other
77.01k stars 45.78k forks source link

Object Detection API - Stucked on step 0 #8308

Closed horczech closed 4 years ago

horczech commented 4 years ago

System information

Please provide the entire URL of the model you are using?

http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz Describe the current behavior The transfer learning gets stuck on step 0 Describe the expected behavior The learning of the model starts. I thought that the problem might be connected with some mistake while creating the train.record and test.record files, but the training works with R-CNN models. It just gets stucked with SSD models. Code to reproduce the issue

I run the following command from the folder object_detection

python model_main.py --model_dir=C:\ObjectDetection\assests\training_output\ssd_mob_v1\out --pipeline_config_path=C:\ObjectDetection\assests\pretrained_models\ssd_mobilenet_v1_coco_2018_01_28\configs\pipeline_v1.config

Vesrion of Tensorflow Object Detection library: commit 6a76ce5b0c33e542f9ec8645dfd340b26f561f17 tag: v1.13.0-1090-g6a76ce5b

Training data can be found here: https://drive.google.com/drive/folders/1akj4vfFmPdA5BZHkArFFSNhpHjeJvotr?usp=sharing

Config file:

model { ssd { num_classes: 8 image_resizer { fixed_shape_resizer { height: 300 width: 300 } } feature_extractor { type: "ssd_mobilenet_v1" depth_multiplier: 1.0 min_depth: 16 conv_hyperparams { regularizer { l2_regularizer { weight: 3.99999989895e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.0299999993294 } } activation: RELU_6 batch_norm { decay: 0.999700009823 center: true scale: true epsilon: 0.0010000000475 train: true } } } box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true } } similarity_calculator { iou_similarity { } } box_predictor { convolutional_box_predictor { conv_hyperparams { regularizer { l2_regularizer { weight: 3.99999989895e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.0299999993294 } } activation: RELU_6 batch_norm { decay: 0.999700009823 center: true scale: true epsilon: 0.0010000000475 train: true } } min_depth: 0 max_depth: 0 num_layers_before_predictor: 0 use_dropout: false dropout_keep_probability: 0.800000011921 kernel_size: 1 box_code_size: 4 apply_sigmoid_to_scores: false } } anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.20000000298 max_scale: 0.949999988079 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.333299994469 } } post_processing { batch_non_max_suppression { score_threshold: 0.300000011921 iou_threshold: 0.600000023842 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } normalize_loss_by_num_matches: true loss { localization_loss { weighted_smooth_l1 { } } classification_loss { weighted_sigmoid { } } hard_example_miner { num_hard_examples: 3000 iou_threshold: 0.990000009537 loss_type: CLASSIFICATION max_negatives_per_positive: 3 min_negatives_per_image: 0 } classification_weight: 1.0 localization_weight: 1.0 } } } train_config { batch_size: 4 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { ssd_random_crop { } } optimizer { rms_prop_optimizer { learning_rate { exponential_decay_learning_rate { initial_learning_rate: 0.00400000018999 decay_steps: 800720 decay_factor: 0.949999988079 } } momentum_optimizer_value: 0.899999976158 decay: 0.899999976158 epsilon: 1.0 } } fine_tune_checkpoint: "C:/ObjectDetection/assests/pretrained_models/ssd_mobilenet_v1_coco_2018_01_28/model.ckpt" from_detection_checkpoint: true num_steps: 200000 } train_input_reader { label_map_path: "C:/ObjectDetection/assests/datasets/dataset_without_bullshit_listitem_classes/label_map.pbtxt" tf_record_input_reader { input_path: "C:/ObjectDetection/assests/datasets/dataset_without_bullshit_listitem_classes/train.record" } } eval_config { num_examples: 200 max_evals: 10 use_moving_averages: false } eval_input_reader { label_map_path: "C:/ObjectDetection/assests/datasets/dataset_without_bullshit_listitem_classes/label_map.pbtxt" shuffle: false num_readers: 1 tf_record_input_reader { input_path: "C:/ObjectDetection/assests/datasets/dataset_without_bullshit_listitem_classes/test.record" } }

Other info / logs

(tf_gpu) C:\ObjectDetection\models\research\object_detection>python model_main.py --model_dir=C:\ObjectDetection\assests\training_output\ssd_mob_v1\out --pipeline_config_path=C:\ObjectDetection\assests\pretrained_models\ssd_mobilenet_v1_coco_2018_01_28\configs\pipeline_v1.config 2020-03-19 00:24:20.869069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

WARNING:tensorflow:From model_main.py:109: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0319 00:24:22.592000 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

W0319 00:24:22.594998 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

WARNING:tensorflow:Forced number of epochs for all eval validations to be 1. W0319 00:24:22.594998 13412 model_lib.py:629] Forced number of epochs for all eval validations to be 1. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0319 00:24:22.595998 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Maybe overwriting train_steps: None I0319 00:24:22.596998 13412 config_util.py:488] Maybe overwriting train_steps: None INFO:tensorflow:Maybe overwriting use_bfloat16: False I0319 00:24:22.596998 13412 config_util.py:488] Maybe overwriting use_bfloat16: False INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1 I0319 00:24:22.597998 13412 config_util.py:488] Maybe overwriting sample_1_of_n_eval_examples: 1 INFO:tensorflow:Maybe overwriting eval_num_epochs: 1 I0319 00:24:22.597998 13412 config_util.py:488] Maybe overwriting eval_num_epochs: 1 INFO:tensorflow:Maybe overwriting load_pretrained: True I0319 00:24:22.599000 13412 config_util.py:488] Maybe overwriting load_pretrained: True INFO:tensorflow:Ignoring config override key: load_pretrained I0319 00:24:22.599000 13412 config_util.py:498] Ignoring config override key: load_pretrained WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered eval_on_train_input_config.num_epochs = 0. Overwriting num_epochs to 1. W0319 00:24:22.599998 13412 model_lib.py:645] Expected number of evaluation epochs is 1, but instead encountered eval_on_train_input_config.num_epochs = 0. Overwriting num_epochs to 1. INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False I0319 00:24:22.599998 13412 model_lib.py:680] create_estimator_and_inputs: use_tpu False, export_to_tpu False INFO:tensorflow:Using config: {'_model_dir': 'C:\ObjectDetection\assests\training_output\ssd_mob_v1\out', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001EA0BEE7C88>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} I0319 00:24:22.603000 13412 estimator.py:212] Using config: {'_model_dir': 'C:\ObjectDetection\assests\training_output\ssd_mob_v1\out', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001EA0BEE7C88>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:Estimator's model_fn (<function create_model_fn..model_fn at 0x000001EA0BEE65E8>) includes params argument, but params are not passed to Estimator. W0319 00:24:22.603000 13412 model_fn.py:630] Estimator's model_fn (<function create_model_fn..model_fn at 0x000001EA0BEE65E8>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Not using Distribute Coordinator. I0319 00:24:22.603999 13412 estimator_training.py:186] Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). I0319 00:24:22.604998 13412 training.py:612] Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. I0319 00:24:22.604998 13412 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W0319 00:24:22.608999 13412 deprecation.py:323] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0319 00:24:22.614000 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W0319 00:24:22.614000 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W0319 00:24:22.622998 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W0319 00:24:22.624999 13412 dataset_builder.py:72] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.experimental.parallel_interleave(...). W0319 00:24:22.627998 13412 deprecation.py:323] From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.experimental.parallel_interleave(...). WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic. W0319 00:24:22.627998 13412 deprecation.py:323] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() W0319 00:24:22.641998 13412 deprecation.py:323] From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Usetf.data.Dataset.map() WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.

W0319 00:24:23.526571 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.

WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

W0319 00:24:28.580229 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0319 00:24:28.641227 13412 deprecation.py:323] From C:\ObjectDetection\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0319 00:24:30.205249 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\operators\control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W0319 00:24:32.649807 13412 api.py:332] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\operators\control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W0319 00:24:34.903855 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

W0319 00:24:34.906855 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W0319 00:24:35.260750 13412 deprecation.py:323] From C:\ObjectDetection\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

W0319 00:24:36.408305 13412 module_wrapper.py:139] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.batch(..., drop_remainder=True). W0319 00:24:36.778973 13412 deprecation.py:323] From C:\ObjectDetection\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.batch(..., drop_remainder=True). INFO:tensorflow:Calling model_fn. I0319 00:24:36.787972 13412 estimator.py:1148] Calling model_fn. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:589: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0319 00:24:36.824971 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:589: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:597: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0319 00:24:36.825971 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:597: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. W0319 00:24:36.826973 13412 deprecation.py:323] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

W0319 00:24:37.904242 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:37.912241 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:37.933770 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:37.956631 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:37.977632 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:37.996631 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0319 00:24:38.017641 13412 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0 WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

W0319 00:24:38.040641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

W0319 00:24:38.042641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:353: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

W0319 00:24:38.046640 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:353: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

W0319 00:24:38.374641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:1163: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0319 00:24:38.664641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:1163: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\core\losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.

W0319 00:24:38.669640 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\core\losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\core\losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.

W0319 00:24:38.670642 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\core\losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:1275: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

W0319 00:24:38.749640 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\meta_architectures\ssd_meta_arch.py:1275: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:380: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0319 00:24:38.753640 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:380: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\utils\learning_schedules.py:66: The name tf.train.exponential_decay is deprecated. Please use tf.compat.v1.train.exponential_decay instead.

W0319 00:24:38.753640 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\utils\learning_schedules.py:66: The name tf.train.exponential_decay is deprecated. Please use tf.compat.v1.train.exponential_decay instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\builders\optimizer_builder.py:47: The name tf.train.RMSPropOptimizer is deprecated. Please use tf.compat.v1.train.RMSPropOptimizer instead.

W0319 00:24:38.759641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\builders\optimizer_builder.py:47: The name tf.train.RMSPropOptimizer is deprecated. Please use tf.compat.v1.train.RMSPropOptimizer instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:398: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

W0319 00:24:38.759641 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:398: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\training\rmsprop.py:119: calling Ones.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0319 00:24:39.390025 13412 deprecation.py:506] From C:\Users\robot\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_core\python\training\rmsprop.py:119: calling Ones.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:515: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0319 00:24:41.535035 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:515: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:519: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

W0319 00:24:41.793184 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:519: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

WARNING:tensorflow:From C:\ObjectDetection\models\research\object_detection\model_lib.py:520: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

W0319 00:24:41.795184 13412 module_wrapper.py:139] From C:\ObjectDetection\models\research\object_detection\model_lib.py:520: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

INFO:tensorflow:Done calling model_fn. I0319 00:24:41.795184 13412 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. I0319 00:24:41.796184 13412 basic_session_run_hooks.py:541] Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. I0319 00:24:43.547633 13412 monitored_session.py:240] Graph was finalized. 2020-03-19 00:24:43.549413: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2020-03-19 00:24:43.554092: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll 2020-03-19 00:24:43.587122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: GeForce RTX 2070 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:2b:00.0 2020-03-19 00:24:43.591128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll 2020-03-19 00:24:43.595288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll 2020-03-19 00:24:43.600594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll 2020-03-19 00:24:43.603542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll 2020-03-19 00:24:43.609042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll 2020-03-19 00:24:43.612936: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll 2020-03-19 00:24:43.621477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2020-03-19 00:24:43.624402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-03-19 00:24:44.114623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-03-19 00:24:44.116580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-03-19 00:24:44.118542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-03-19 00:24:44.120620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6283 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:2b:00.0, compute capability: 7.5) INFO:tensorflow:Running local_init_op. I0319 00:24:46.515733 13412 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0319 00:24:46.779241 13412 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into C:\ObjectDetection\assests\training_output\ssd_mob_v1\out\model.ckpt. I0319 00:24:51.621353 13412 basic_session_run_hooks.py:606] Saving checkpoints for 0 into C:\ObjectDetection\assests\training_output\ssd_mob_v1\out\model.ckpt. 2020-03-19 00:24:57.026117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2020-03-19 00:24:58.299653: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows Relying on driver to perform ptx compilation. This message will be only logged once. 2020-03-19 00:24:58.332897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll INFO:tensorflow:loss = 44.150455, step = 0 I0319 00:25:00.139314 13412 basic_session_run_hooks.py:262] loss = 44.150455, step = 0

horczech commented 4 years ago

Ok, problem solved while writing this issue. I realized that I have tensorflow_gpu-1.15.0 which is not tested according to this table so I have downgraded to tensorflow_gpu-1.14.0 and now it is working.

Sorry for troubles