Open Totemi1324 opened 4 years ago
Rajat-Mehta
commented
4 years ago @Totemi1324 did you figure out the solution? I am also stuck with this issue and I tried many solutions proposed by others but nothing worked so far. Any idea how to solve this ?
Totemi1324
commented
4 years ago @Rajat-Mehta I posted this issue on several threads and still didn't earn a specific answer, so I can't really tell you what is wrong. But by accident, I discovered how to fix it. For me, it helped to set my batch size in the .config-file to 2 (since previously, it was set to 1). Therefore, I assume that yet another time, this is bad exception handling of Tensorflow and your batch size must be an even number or else some functions don't work properly. I'd recommend you to just experiment with the batch size to find the one that is the largest your system can handle (if Tensorflow fails to allocate enough memory, it will tell you in advance) and is even. Good luck!
opditextest
commented
4 years ago @Rajat-Mehta @Totemi1324 @tombstone @jch1 @pkulzc
I am training a custom ssd_mobilenet_v2_quantized_300x300 TensorFlow model for object detection using Google Colab with the downgraded version of TensorFlow 1.15.2 because I use to train my model on previous version of TensorFlow i.e. 1.14.0 but due to the latest update to version 2.2.0, I get the strange errors and therefore I can't use the latest version. Using 1.15.2 version and selection even batch size, I successfully starts the training process but after some time, the training process stops with the following errors.
TypeError: 'numpy.float64' object cannot be interpreted as an integer
self._traceback = tf_stack.extract_stack()
Complete training log can be seen as follows;
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0619 08:34:55.917273 140679256610688 model_lib.py:717] Forced number of epochs for all eval validations to be 1.
INFO:tensorflow:Maybe overwriting train_steps: 50000
I0619 08:34:55.917532 140679256610688 config_util.py:523] Maybe overwriting train_steps: 50000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0619 08:34:55.917701 140679256610688 config_util.py:523] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0619 08:34:55.917855 140679256610688 config_util.py:523] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0619 08:34:55.917991 140679256610688 config_util.py:523] Maybe overwriting eval_num_epochs: 1
INFO:tensorflow:Maybe overwriting load_pretrained: True
I0619 08:34:55.918123 140679256610688 config_util.py:523] Maybe overwriting load_pretrained: True
INFO:tensorflow:Ignoring config override key: load_pretrained
I0619 08:34:55.918251 140679256610688 config_util.py:533] Ignoring config override key: load_pretrained
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W0619 08:34:55.919228 140679256610688 model_lib.py:733] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False
I0619 08:34:55.919431 140679256610688 model_lib.py:768] create_estimator_and_inputs: use_tpu False, export_to_tpu False
INFO:tensorflow:Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0619 08:34:55.919958 140679256610688 estimator.py:212] Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator.
W0619 08:34:55.920258 140679256610688 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0619 08:34:55.921080 140679256610688 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0619 08:34:55.921281 140679256610688 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0619 08:34:55.921539 140679256610688 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0619 08:34:55.935507 140679256610688 deprecation.py:323] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0619 08:34:55.974960 140679256610688 dataset_builder.py:83] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0619 08:34:55.981717 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0619 08:34:56.005888 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'
W0619 08:34:56.041087 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'
WARNING:tensorflow:Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4
W0619 08:34:56.271473 140679256610688 ag_logging.py:146] Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0619 08:34:56.278620 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0619 08:34:56.287626 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0619 08:34:56.404175 140679256610688 deprecation.py:323] From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0619 08:34:57.319203 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
INFO:tensorflow:Calling model_fn.
I0619 08:34:58.000243 140679256610688 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0619 08:34:58.086804 140679256610688 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.276282 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.313794 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.349460 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.385352 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.422045 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:35:01.460603 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold
I0619 08:35:12.419386 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold
I0619 08:35:12.419960 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold
I0619 08:35:12.420622 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold
I0619 08:35:12.421021 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold
I0619 08:35:12.421649 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold
I0619 08:35:12.422075 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold
I0619 08:35:12.422821 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold
I0619 08:35:12.423317 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold
I0619 08:35:12.423930 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold
I0619 08:35:12.424316 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold
I0619 08:35:12.425071 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold
I0619 08:35:12.425590 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold
I0619 08:35:12.426191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold
I0619 08:35:12.426571 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold
I0619 08:35:12.427229 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold
I0619 08:35:12.427820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold
I0619 08:35:12.428410 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold
I0619 08:35:12.428805 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold
I0619 08:35:12.429387 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold
I0619 08:35:12.429798 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold
I0619 08:35:12.430403 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold
I0619 08:35:12.430883 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold
I0619 08:35:12.431473 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold
I0619 08:35:12.431915 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold
I0619 08:35:12.432562 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold
I0619 08:35:12.433037 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold
I0619 08:35:12.433677 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold
I0619 08:35:12.434067 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold
I0619 08:35:12.434737 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold
I0619 08:35:12.435177 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold
I0619 08:35:12.435865 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold
I0619 08:35:12.436269 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold
I0619 08:35:12.436948 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold
I0619 08:35:12.437350 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold
I0619 08:35:12.438012 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold
I0619 08:35:12.438425 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold
I0619 08:35:12.438820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold
I0619 08:35:12.439191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold
I0619 08:35:12.439666 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold
I0619 08:35:12.440045 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold
I0619 08:35:12.440409 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold
I0619 08:35:12.440825 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold
I0619 08:35:12.441197 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0619 08:35:22.078743 140679256610688 deprecation.py:506] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
I0619 08:35:30.469601 140679256610688 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0619 08:35:30.471154 140679256610688 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0619 08:35:37.414344 140679256610688 monitored_session.py:240] Graph was finalized.
2020-06-19 08:35:37.428439: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2020-06-19 08:35:37.429078: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-19 08:35:37.429122: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-19 08:35:37.434521: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-19 08:35:37.613365: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:37.614447: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-19 08:35:37.614487: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2020-06-19 08:35:37.615832: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:37.616699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-06-19 08:35:37.617137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-19 08:35:37.869500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-19 08:35:37.990427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-06-19 08:35:38.025559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-06-19 08:35:38.280571: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-06-19 08:35:38.323569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-06-19 08:35:38.816387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-19 08:35:38.816668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:38.817676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:38.818508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-19 08:35:38.822059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-19 08:35:38.823827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-19 08:35:38.823860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-19 08:35:38.823871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-19 08:35:38.825211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:38.826359: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:35:38.827380: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-06-19 08:35:38.827455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
INFO:tensorflow:Running local_init_op.
I0619 08:35:51.728541 140679256610688 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0619 08:35:52.421987 140679256610688 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into training/model.ckpt.
I0619 08:36:11.138946 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 0 into training/model.ckpt.
2020-06-19 08:36:32.110483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-19 08:36:36.543281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
INFO:tensorflow:loss = 20.36485, step = 0
I0619 08:36:38.734370 140679256610688 basic_session_run_hooks.py:262] loss = 20.36485, step = 0
INFO:tensorflow:global_step/sec: 2.70978
I0619 08:37:15.636997 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 2.70978
INFO:tensorflow:loss = 2.5266151, step = 100 (36.904 sec)
I0619 08:37:15.637996 140679256610688 basic_session_run_hooks.py:260] loss = 2.5266151, step = 100 (36.904 sec)
INFO:tensorflow:global_step/sec: 4.4168
I0619 08:37:38.277826 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4168
INFO:tensorflow:loss = 3.0113943, step = 200 (22.641 sec)
I0619 08:37:38.278793 140679256610688 basic_session_run_hooks.py:260] loss = 3.0113943, step = 200 (22.641 sec)
INFO:tensorflow:global_step/sec: 4.44349
I0619 08:38:00.782637 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44349
INFO:tensorflow:loss = 1.3504767, step = 300 (22.505 sec)
I0619 08:38:00.783768 140679256610688 basic_session_run_hooks.py:260] loss = 1.3504767, step = 300 (22.505 sec)
INFO:tensorflow:global_step/sec: 4.43355
I0619 08:38:23.337961 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43355
INFO:tensorflow:loss = 2.7894356, step = 400 (22.555 sec)
I0619 08:38:23.339154 140679256610688 basic_session_run_hooks.py:260] loss = 2.7894356, step = 400 (22.555 sec)
INFO:tensorflow:global_step/sec: 4.43814
I0619 08:38:45.869889 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43814
INFO:tensorflow:loss = 6.171316, step = 500 (22.532 sec)
I0619 08:38:45.870794 140679256610688 basic_session_run_hooks.py:260] loss = 6.171316, step = 500 (22.532 sec)
INFO:tensorflow:global_step/sec: 4.41364
I0619 08:39:08.526993 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41364
INFO:tensorflow:loss = 2.9929922, step = 600 (22.657 sec)
I0619 08:39:08.528079 140679256610688 basic_session_run_hooks.py:260] loss = 2.9929922, step = 600 (22.657 sec)
INFO:tensorflow:global_step/sec: 4.44947
I0619 08:39:31.001498 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44947
INFO:tensorflow:loss = 4.0605145, step = 700 (22.474 sec)
I0619 08:39:31.002526 140679256610688 basic_session_run_hooks.py:260] loss = 4.0605145, step = 700 (22.474 sec)
INFO:tensorflow:global_step/sec: 4.43146
I0619 08:39:53.567426 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43146
INFO:tensorflow:loss = 3.246991, step = 800 (22.568 sec)
I0619 08:39:53.570216 140679256610688 basic_session_run_hooks.py:260] loss = 3.246991, step = 800 (22.568 sec)
INFO:tensorflow:global_step/sec: 4.43037
I0619 08:40:16.138887 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43037
INFO:tensorflow:loss = 3.6994023, step = 900 (22.570 sec)
I0619 08:40:16.139893 140679256610688 basic_session_run_hooks.py:260] loss = 3.6994023, step = 900 (22.570 sec)
INFO:tensorflow:global_step/sec: 4.44706
I0619 08:40:38.625646 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44706
INFO:tensorflow:loss = 3.306572, step = 1000 (22.487 sec)
I0619 08:40:38.626590 140679256610688 basic_session_run_hooks.py:260] loss = 3.306572, step = 1000 (22.487 sec)
INFO:tensorflow:global_step/sec: 4.42377
I0619 08:41:01.230800 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.42377
INFO:tensorflow:loss = 1.8826125, step = 1100 (22.605 sec)
I0619 08:41:01.231729 140679256610688 basic_session_run_hooks.py:260] loss = 1.8826125, step = 1100 (22.605 sec)
INFO:tensorflow:global_step/sec: 4.439
I0619 08:41:23.758457 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.439
INFO:tensorflow:loss = 2.5373318, step = 1200 (22.528 sec)
I0619 08:41:23.759881 140679256610688 basic_session_run_hooks.py:260] loss = 2.5373318, step = 1200 (22.528 sec)
INFO:tensorflow:global_step/sec: 4.43686
I0619 08:41:46.296886 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43686
INFO:tensorflow:loss = 3.7123256, step = 1300 (22.538 sec)
I0619 08:41:46.297836 140679256610688 basic_session_run_hooks.py:260] loss = 3.7123256, step = 1300 (22.538 sec)
INFO:tensorflow:global_step/sec: 4.37992
I0619 08:42:09.128318 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.37992
INFO:tensorflow:loss = 2.2112222, step = 1400 (22.832 sec)
I0619 08:42:09.129705 140679256610688 basic_session_run_hooks.py:260] loss = 2.2112222, step = 1400 (22.832 sec)
INFO:tensorflow:global_step/sec: 4.44905
I0619 08:42:31.605058 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44905
INFO:tensorflow:loss = 1.3892399, step = 1500 (22.476 sec)
I0619 08:42:31.606064 140679256610688 basic_session_run_hooks.py:260] loss = 1.3892399, step = 1500 (22.476 sec)
INFO:tensorflow:global_step/sec: 4.45304
I0619 08:42:54.061597 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.45304
INFO:tensorflow:loss = 2.1145034, step = 1600 (22.457 sec)
I0619 08:42:54.062722 140679256610688 basic_session_run_hooks.py:260] loss = 2.1145034, step = 1600 (22.457 sec)
INFO:tensorflow:global_step/sec: 4.43729
I0619 08:43:16.597952 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43729
INFO:tensorflow:loss = 1.0390191, step = 1700 (22.536 sec)
I0619 08:43:16.598960 140679256610688 basic_session_run_hooks.py:260] loss = 1.0390191, step = 1700 (22.536 sec)
INFO:tensorflow:global_step/sec: 4.41662
I0619 08:43:39.239702 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41662
INFO:tensorflow:loss = 3.100757, step = 1800 (22.642 sec)
I0619 08:43:39.241185 140679256610688 basic_session_run_hooks.py:260] loss = 3.100757, step = 1800 (22.642 sec)
INFO:tensorflow:global_step/sec: 4.39809
I0619 08:44:01.976768 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.39809
INFO:tensorflow:loss = 3.6370416, step = 1900 (22.737 sec)
I0619 08:44:01.977803 140679256610688 basic_session_run_hooks.py:260] loss = 3.6370416, step = 1900 (22.737 sec)
INFO:tensorflow:global_step/sec: 4.43766
I0619 08:44:24.511184 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766
INFO:tensorflow:loss = 2.2865329, step = 2000 (22.535 sec)
I0619 08:44:24.512323 140679256610688 basic_session_run_hooks.py:260] loss = 2.2865329, step = 2000 (22.535 sec)
INFO:tensorflow:global_step/sec: 4.41886
I0619 08:44:47.141552 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41886
INFO:tensorflow:loss = 2.2199974, step = 2100 (22.630 sec)
I0619 08:44:47.142618 140679256610688 basic_session_run_hooks.py:260] loss = 2.2199974, step = 2100 (22.630 sec)
INFO:tensorflow:global_step/sec: 4.44154
I0619 08:45:09.656170 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44154
INFO:tensorflow:loss = 2.1248786, step = 2200 (22.515 sec)
I0619 08:45:09.657206 140679256610688 basic_session_run_hooks.py:260] loss = 2.1248786, step = 2200 (22.515 sec)
INFO:tensorflow:global_step/sec: 4.43766
I0619 08:45:32.190551 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766
INFO:tensorflow:loss = 4.8454585, step = 2300 (22.535 sec)
I0619 08:45:32.191816 140679256610688 basic_session_run_hooks.py:260] loss = 4.8454585, step = 2300 (22.535 sec)
INFO:tensorflow:global_step/sec: 4.4216
I0619 08:45:54.806908 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4216
INFO:tensorflow:loss = 0.9450711, step = 2400 (22.616 sec)
I0619 08:45:54.808129 140679256610688 basic_session_run_hooks.py:260] loss = 0.9450711, step = 2400 (22.616 sec)
INFO:tensorflow:Saving checkpoints for 2492 into training/model.ckpt.
I0619 08:46:15.349621 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 2492 into training/model.ckpt.
WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'
W0619 08:46:18.131156 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'
WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4
W0619 08:46:18.347941 140679256610688 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4
INFO:tensorflow:Calling model_fn.
I0619 08:46:18.968489 140679256610688 estimator.py:1148] Calling model_fn.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.590440 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.626965 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.665331 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.702586 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.738639 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0619 08:46:21.774662 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold
I0619 08:46:24.474975 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold
I0619 08:46:24.475332 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold
I0619 08:46:24.475668 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold
I0619 08:46:24.475931 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold
I0619 08:46:24.476265 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold
I0619 08:46:24.476573 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold
I0619 08:46:24.476926 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold
I0619 08:46:24.477203 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold
I0619 08:46:24.477541 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold
I0619 08:46:24.477811 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold
I0619 08:46:24.478186 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold
I0619 08:46:24.478434 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold
I0619 08:46:24.478747 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold
I0619 08:46:24.478998 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold
I0619 08:46:24.479341 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold
I0619 08:46:24.479565 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold
I0619 08:46:24.479938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold
I0619 08:46:24.480194 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold
I0619 08:46:24.480589 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold
I0619 08:46:24.480898 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold
I0619 08:46:24.481264 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold
I0619 08:46:24.481486 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold
I0619 08:46:24.481839 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold
I0619 08:46:24.482104 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold
I0619 08:46:24.482481 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold
I0619 08:46:24.482767 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold
I0619 08:46:24.483124 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold
I0619 08:46:24.483347 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold
I0619 08:46:24.483694 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold
I0619 08:46:24.483980 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold
I0619 08:46:24.484319 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold
I0619 08:46:24.484550 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold
I0619 08:46:24.484876 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold
I0619 08:46:24.485111 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold
I0619 08:46:24.485412 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold
I0619 08:46:24.485624 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold
I0619 08:46:24.485938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold
I0619 08:46:24.486167 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold
I0619 08:46:24.486395 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold
I0619 08:46:24.486675 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold
I0619 08:46:24.486938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold
I0619 08:46:24.487241 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold
INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold
I0619 08:46:24.487478 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold
WARNING:tensorflow:From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0619 08:46:24.518594 140679256610688 deprecation.py:323] From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0619 08:46:24.766963 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
INFO:tensorflow:Done calling model_fn.
I0619 08:46:25.693374 140679256610688 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-06-19T08:46:25Z
I0619 08:46:25.713920 140679256610688 evaluation.py:255] Starting evaluation at 2020-06-19T08:46:25Z
INFO:tensorflow:Graph was finalized.
I0619 08:46:26.684044 140679256610688 monitored_session.py:240] Graph was finalized.
2020-06-19 08:46:26.685194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:46:26.685930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-06-19 08:46:26.686050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-19 08:46:26.686083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-19 08:46:26.686112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-06-19 08:46:26.686141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-06-19 08:46:26.686169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-06-19 08:46:26.686195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-06-19 08:46:26.686228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-19 08:46:26.686329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:46:26.687075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:46:26.687713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-19 08:46:26.687792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-19 08:46:26.687810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-19 08:46:26.687819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-19 08:46:26.687940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:46:26.688724: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-19 08:46:26.689491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
INFO:tensorflow:Restoring parameters from training/model.ckpt-2492
I0619 08:46:26.690679 140679256610688 saver.py:1284] Restoring parameters from training/model.ckpt-2492
INFO:tensorflow:Running local_init_op.
I0619 08:46:28.285569 140679256610688 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0619 08:46:28.471102 140679256610688 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 2 images.
I0619 08:46:31.930884 140675338995456 coco_evaluation.py:237] Performing evaluation on 2 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0619 08:46:31.931652 140675338995456 coco_tools.py:116] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0619 08:46:31.932090 140675338995456 coco_tools.py:138] DONE (t=0.00s)
creating index...
index created!
2020-06-19 08:46:31.949144: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
ret = func(*args)
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func
self._metrics = self.evaluate()
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__
cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type)
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__
self.setDetParams()
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
(1) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once
session.run(eval_ops, feed_dict)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Out of range: End of sequence
[[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'IteratorGetNext':
File "content/models/research/object_detection/model_main.py", line 114, in <module>
tf.app.run()
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "content/models/research/object_detection/model_main.py", line 110, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
return executor.run()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run
run_metadata=run_metadata))
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run
if self._save(run_context.session, global_step):
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save
if l.after_save(session, step):
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
self._evaluate(global_step_value) # updates self.eval_result
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
self._evaluator.evaluate_and_export())
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
return _evaluate()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate
self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph
self._call_model_fn_eval(input_fn, self.config))
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1544, in _call_model_fn_eval
input_fn, ModeKeys.EVAL)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn
self._call_input_fn(input_fn, mode))
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/util.py", line 65, in parse_input_fn_result
result = iterator.get_next()
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
output_shapes=output_shapes, name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
ret = func(*args)
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func
self._metrics = self.evaluate()
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__
cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type)
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__
self.setDetParams()
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
[[{{node PyFunc_3}}]]
[[cond/Detections_Left_Groundtruth_Right/0/_4927]]
(1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
ret = func(*args)
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func
self._metrics = self.evaluate()
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__
cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type)
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__
self.setDetParams()
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
[[{{node PyFunc_3}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/models/research/object_detection/model_main.py", line 114, in <module>
tf.app.run()
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/content/models/research/object_detection/model_main.py", line 110, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
return executor.run()
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
saving_listeners)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
raise value
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run
run_metadata=run_metadata))
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run
if self._save(run_context.session, global_step):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save
if l.after_save(session, step):
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
self._evaluate(global_step_value) # updates self.eval_result
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
self._evaluator.evaluate_and_export())
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
name=name)
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
return _evaluate()
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 511, in _evaluate
output_dir=self.eval_dir(name))
File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1619, in _evaluate_run
config=self._session_config)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once
session.run(eval_ops, feed_dict)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 861, in __exit__
self._close_internal(exception_type)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 894, in _close_internal
h.end(self._coordinated_creator.tf_sess)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 951, in end
self._final_ops, feed_dict=self._final_ops_feed_dict)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
ret = func(*args)
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func
self._metrics = self.evaluate()
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__
cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type)
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__
self.setDetParams()
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
[[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
[[cond/Detections_Left_Groundtruth_Right/0/_4927]]
(1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
ret = func(*args)
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func
self._metrics = self.evaluate()
File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__
cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type)
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__
self.setDetParams()
File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
[[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'PyFunc_3':
File "content/models/research/object_detection/model_main.py", line 114, in <module>
tf.app.run()
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "content/models/research/object_detection/model_main.py", line 110, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
return executor.run()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
saving_listeners)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run
run_metadata=run_metadata))
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run
if self._save(run_context.session, global_step):
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save
if l.after_save(session, step):
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
self._evaluate(global_step_value) # updates self.eval_result
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
self._evaluator.evaluate_and_export())
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
return _evaluate()
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate
self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph
self._call_model_fn_eval(input_fn, self.config))
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1547, in _call_model_fn_eval
features, labels, ModeKeys.EVAL, config)
File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "content/models/research/object_detection/model_lib.py", line 570, in model_fn
eval_config, list(category_index.values()), eval_dict)
File "content/models/research/object_detection/eval_util.py", line 1045, in get_eval_metric_ops_for_evaluators
eval_dict))
File "content/models/research/object_detection/metrics/coco_evaluation.py", line 426, in get_estimator_eval_metric_ops
first_value_op = tf.py_func(first_value_func, [], tf.float32)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 513, in py_func
return py_func_common(func, inp, Tout, stateful, name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 495, in py_func_common
func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 318, in _internal_py_func
input=inp, token=token, Tout=Tout, name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_script_ops.py", line 170, in py_func
"PyFunc", input=input, token=token, Tout=Tout, name=name)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()```
What are the possibilites of getting over this issue, any kind of recommendations?
ttducqt
commented
4 years ago @Rajat-Mehta @Totemi1324 @tombstone @jch1 @pkulzc I am training a custom ssd_mobilenet_v2_quantized_300x300 TensorFlow model for object detection using Google Colab with the downgraded version of TensorFlow 1.15.2 because I use to train my model on previous version of TensorFlow i.e. 1.14.0 but due to the latest update to version 2.2.0, I get the strange errors and therefore I can't use the latest version. Using 1.15.2 version and selection even batch size, I successfully starts the training process but after some time, the training process stops with the following errors.
TypeError: 'numpy.float64' object cannot be interpreted as an integerself._traceback = tf_stack.extract_stack()Complete training log can be seen as follows;
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1. W0619 08:34:55.917273 140679256610688 model_lib.py:717] Forced number of epochs for all eval validations to be 1. INFO:tensorflow:Maybe overwriting train_steps: 50000 I0619 08:34:55.917532 140679256610688 config_util.py:523] Maybe overwriting train_steps: 50000 INFO:tensorflow:Maybe overwriting use_bfloat16: False I0619 08:34:55.917701 140679256610688 config_util.py:523] Maybe overwriting use_bfloat16: False INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1 I0619 08:34:55.917855 140679256610688 config_util.py:523] Maybe overwriting sample_1_of_n_eval_examples: 1 INFO:tensorflow:Maybe overwriting eval_num_epochs: 1 I0619 08:34:55.917991 140679256610688 config_util.py:523] Maybe overwriting eval_num_epochs: 1 INFO:tensorflow:Maybe overwriting load_pretrained: True I0619 08:34:55.918123 140679256610688 config_util.py:523] Maybe overwriting load_pretrained: True INFO:tensorflow:Ignoring config override key: load_pretrained I0619 08:34:55.918251 140679256610688 config_util.py:533] Ignoring config override key: load_pretrained WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. W0619 08:34:55.919228 140679256610688 model_lib.py:733] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False I0619 08:34:55.919431 140679256610688 model_lib.py:768] create_estimator_and_inputs: use_tpu False, export_to_tpu False INFO:tensorflow:Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} I0619 08:34:55.919958 140679256610688 estimator.py:212] Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator. W0619 08:34:55.920258 140679256610688 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Not using Distribute Coordinator. I0619 08:34:55.921080 140679256610688 estimator_training.py:186] Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). I0619 08:34:55.921281 140679256610688 training.py:612] Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. I0619 08:34:55.921539 140679256610688 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W0619 08:34:55.935507 140679256610688 deprecation.py:323] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W0619 08:34:55.974960 140679256610688 dataset_builder.py:83] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. W0619 08:34:55.981717 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() W0619 08:34:56.005888 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' W0619 08:34:56.041087 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' WARNING:tensorflow:Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 W0619 08:34:56.271473 140679256610688 ag_logging.py:146] Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. W0619 08:34:56.278620 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. WARNING:tensorflow:From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0619 08:34:56.287626 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W0619 08:34:56.404175 140679256610688 deprecation.py:323] From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0619 08:34:57.319203 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. INFO:tensorflow:Calling model_fn. I0619 08:34:58.000243 140679256610688 estimator.py:1148] Calling model_fn. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. W0619 08:34:58.086804 140679256610688 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.276282 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.313794 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.349460 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.385352 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.422045 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.460603 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold I0619 08:35:12.419386 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold I0619 08:35:12.419960 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold I0619 08:35:12.420622 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold I0619 08:35:12.421021 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold I0619 08:35:12.421649 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold I0619 08:35:12.422075 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold I0619 08:35:12.422821 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold I0619 08:35:12.423317 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold I0619 08:35:12.423930 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold I0619 08:35:12.424316 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold I0619 08:35:12.425071 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold I0619 08:35:12.425590 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold I0619 08:35:12.426191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold I0619 08:35:12.426571 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold I0619 08:35:12.427229 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold I0619 08:35:12.427820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold I0619 08:35:12.428410 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold I0619 08:35:12.428805 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold I0619 08:35:12.429387 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold I0619 08:35:12.429798 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold I0619 08:35:12.430403 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold I0619 08:35:12.430883 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold I0619 08:35:12.431473 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold I0619 08:35:12.431915 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold I0619 08:35:12.432562 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold I0619 08:35:12.433037 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold I0619 08:35:12.433677 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold I0619 08:35:12.434067 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold I0619 08:35:12.434737 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold I0619 08:35:12.435177 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold I0619 08:35:12.435865 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold I0619 08:35:12.436269 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold I0619 08:35:12.436948 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold I0619 08:35:12.437350 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold I0619 08:35:12.438012 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold I0619 08:35:12.438425 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold I0619 08:35:12.438820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold I0619 08:35:12.439191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold I0619 08:35:12.439666 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold I0619 08:35:12.440045 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold I0619 08:35:12.440409 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold I0619 08:35:12.440825 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold I0619 08:35:12.441197 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0619 08:35:22.078743 140679256610688 deprecation.py:506] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. I0619 08:35:30.469601 140679256610688 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. I0619 08:35:30.471154 140679256610688 basic_session_run_hooks.py:541] Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. I0619 08:35:37.414344 140679256610688 monitored_session.py:240] Graph was finalized. 2020-06-19 08:35:37.428439: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz 2020-06-19 08:35:37.429078: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-19 08:35:37.429122: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-06-19 08:35:37.434521: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-06-19 08:35:37.613365: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:37.614447: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-06-19 08:35:37.614487: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2020-06-19 08:35:37.615832: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:37.616699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2020-06-19 08:35:37.617137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:35:37.869500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-06-19 08:35:37.990427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-06-19 08:35:38.025559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-06-19 08:35:38.280571: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-06-19 08:35:38.323569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-06-19 08:35:38.816387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:35:38.816668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.817676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.818508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2020-06-19 08:35:38.822059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:35:38.823827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-06-19 08:35:38.823860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2020-06-19 08:35:38.823871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2020-06-19 08:35:38.825211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.826359: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.827380: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2020-06-19 08:35:38.827455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) INFO:tensorflow:Running local_init_op. I0619 08:35:51.728541 140679256610688 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0619 08:35:52.421987 140679256610688 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into training/model.ckpt. I0619 08:36:11.138946 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 0 into training/model.ckpt. 2020-06-19 08:36:32.110483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:36:36.543281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 INFO:tensorflow:loss = 20.36485, step = 0 I0619 08:36:38.734370 140679256610688 basic_session_run_hooks.py:262] loss = 20.36485, step = 0 INFO:tensorflow:global_step/sec: 2.70978 I0619 08:37:15.636997 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 2.70978 INFO:tensorflow:loss = 2.5266151, step = 100 (36.904 sec) I0619 08:37:15.637996 140679256610688 basic_session_run_hooks.py:260] loss = 2.5266151, step = 100 (36.904 sec) INFO:tensorflow:global_step/sec: 4.4168 I0619 08:37:38.277826 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4168 INFO:tensorflow:loss = 3.0113943, step = 200 (22.641 sec) I0619 08:37:38.278793 140679256610688 basic_session_run_hooks.py:260] loss = 3.0113943, step = 200 (22.641 sec) INFO:tensorflow:global_step/sec: 4.44349 I0619 08:38:00.782637 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44349 INFO:tensorflow:loss = 1.3504767, step = 300 (22.505 sec) I0619 08:38:00.783768 140679256610688 basic_session_run_hooks.py:260] loss = 1.3504767, step = 300 (22.505 sec) INFO:tensorflow:global_step/sec: 4.43355 I0619 08:38:23.337961 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43355 INFO:tensorflow:loss = 2.7894356, step = 400 (22.555 sec) I0619 08:38:23.339154 140679256610688 basic_session_run_hooks.py:260] loss = 2.7894356, step = 400 (22.555 sec) INFO:tensorflow:global_step/sec: 4.43814 I0619 08:38:45.869889 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43814 INFO:tensorflow:loss = 6.171316, step = 500 (22.532 sec) I0619 08:38:45.870794 140679256610688 basic_session_run_hooks.py:260] loss = 6.171316, step = 500 (22.532 sec) INFO:tensorflow:global_step/sec: 4.41364 I0619 08:39:08.526993 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41364 INFO:tensorflow:loss = 2.9929922, step = 600 (22.657 sec) I0619 08:39:08.528079 140679256610688 basic_session_run_hooks.py:260] loss = 2.9929922, step = 600 (22.657 sec) INFO:tensorflow:global_step/sec: 4.44947 I0619 08:39:31.001498 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44947 INFO:tensorflow:loss = 4.0605145, step = 700 (22.474 sec) I0619 08:39:31.002526 140679256610688 basic_session_run_hooks.py:260] loss = 4.0605145, step = 700 (22.474 sec) INFO:tensorflow:global_step/sec: 4.43146 I0619 08:39:53.567426 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43146 INFO:tensorflow:loss = 3.246991, step = 800 (22.568 sec) I0619 08:39:53.570216 140679256610688 basic_session_run_hooks.py:260] loss = 3.246991, step = 800 (22.568 sec) INFO:tensorflow:global_step/sec: 4.43037 I0619 08:40:16.138887 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43037 INFO:tensorflow:loss = 3.6994023, step = 900 (22.570 sec) I0619 08:40:16.139893 140679256610688 basic_session_run_hooks.py:260] loss = 3.6994023, step = 900 (22.570 sec) INFO:tensorflow:global_step/sec: 4.44706 I0619 08:40:38.625646 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44706 INFO:tensorflow:loss = 3.306572, step = 1000 (22.487 sec) I0619 08:40:38.626590 140679256610688 basic_session_run_hooks.py:260] loss = 3.306572, step = 1000 (22.487 sec) INFO:tensorflow:global_step/sec: 4.42377 I0619 08:41:01.230800 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.42377 INFO:tensorflow:loss = 1.8826125, step = 1100 (22.605 sec) I0619 08:41:01.231729 140679256610688 basic_session_run_hooks.py:260] loss = 1.8826125, step = 1100 (22.605 sec) INFO:tensorflow:global_step/sec: 4.439 I0619 08:41:23.758457 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.439 INFO:tensorflow:loss = 2.5373318, step = 1200 (22.528 sec) I0619 08:41:23.759881 140679256610688 basic_session_run_hooks.py:260] loss = 2.5373318, step = 1200 (22.528 sec) INFO:tensorflow:global_step/sec: 4.43686 I0619 08:41:46.296886 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43686 INFO:tensorflow:loss = 3.7123256, step = 1300 (22.538 sec) I0619 08:41:46.297836 140679256610688 basic_session_run_hooks.py:260] loss = 3.7123256, step = 1300 (22.538 sec) INFO:tensorflow:global_step/sec: 4.37992 I0619 08:42:09.128318 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.37992 INFO:tensorflow:loss = 2.2112222, step = 1400 (22.832 sec) I0619 08:42:09.129705 140679256610688 basic_session_run_hooks.py:260] loss = 2.2112222, step = 1400 (22.832 sec) INFO:tensorflow:global_step/sec: 4.44905 I0619 08:42:31.605058 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44905 INFO:tensorflow:loss = 1.3892399, step = 1500 (22.476 sec) I0619 08:42:31.606064 140679256610688 basic_session_run_hooks.py:260] loss = 1.3892399, step = 1500 (22.476 sec) INFO:tensorflow:global_step/sec: 4.45304 I0619 08:42:54.061597 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.45304 INFO:tensorflow:loss = 2.1145034, step = 1600 (22.457 sec) I0619 08:42:54.062722 140679256610688 basic_session_run_hooks.py:260] loss = 2.1145034, step = 1600 (22.457 sec) INFO:tensorflow:global_step/sec: 4.43729 I0619 08:43:16.597952 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43729 INFO:tensorflow:loss = 1.0390191, step = 1700 (22.536 sec) I0619 08:43:16.598960 140679256610688 basic_session_run_hooks.py:260] loss = 1.0390191, step = 1700 (22.536 sec) INFO:tensorflow:global_step/sec: 4.41662 I0619 08:43:39.239702 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41662 INFO:tensorflow:loss = 3.100757, step = 1800 (22.642 sec) I0619 08:43:39.241185 140679256610688 basic_session_run_hooks.py:260] loss = 3.100757, step = 1800 (22.642 sec) INFO:tensorflow:global_step/sec: 4.39809 I0619 08:44:01.976768 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.39809 INFO:tensorflow:loss = 3.6370416, step = 1900 (22.737 sec) I0619 08:44:01.977803 140679256610688 basic_session_run_hooks.py:260] loss = 3.6370416, step = 1900 (22.737 sec) INFO:tensorflow:global_step/sec: 4.43766 I0619 08:44:24.511184 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766 INFO:tensorflow:loss = 2.2865329, step = 2000 (22.535 sec) I0619 08:44:24.512323 140679256610688 basic_session_run_hooks.py:260] loss = 2.2865329, step = 2000 (22.535 sec) INFO:tensorflow:global_step/sec: 4.41886 I0619 08:44:47.141552 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41886 INFO:tensorflow:loss = 2.2199974, step = 2100 (22.630 sec) I0619 08:44:47.142618 140679256610688 basic_session_run_hooks.py:260] loss = 2.2199974, step = 2100 (22.630 sec) INFO:tensorflow:global_step/sec: 4.44154 I0619 08:45:09.656170 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44154 INFO:tensorflow:loss = 2.1248786, step = 2200 (22.515 sec) I0619 08:45:09.657206 140679256610688 basic_session_run_hooks.py:260] loss = 2.1248786, step = 2200 (22.515 sec) INFO:tensorflow:global_step/sec: 4.43766 I0619 08:45:32.190551 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766 INFO:tensorflow:loss = 4.8454585, step = 2300 (22.535 sec) I0619 08:45:32.191816 140679256610688 basic_session_run_hooks.py:260] loss = 4.8454585, step = 2300 (22.535 sec) INFO:tensorflow:global_step/sec: 4.4216 I0619 08:45:54.806908 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4216 INFO:tensorflow:loss = 0.9450711, step = 2400 (22.616 sec) I0619 08:45:54.808129 140679256610688 basic_session_run_hooks.py:260] loss = 0.9450711, step = 2400 (22.616 sec) INFO:tensorflow:Saving checkpoints for 2492 into training/model.ckpt. I0619 08:46:15.349621 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 2492 into training/model.ckpt. WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' W0619 08:46:18.131156 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 W0619 08:46:18.347941 140679256610688 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 INFO:tensorflow:Calling model_fn. I0619 08:46:18.968489 140679256610688 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.590440 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.626965 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.665331 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.702586 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.738639 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.774662 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold I0619 08:46:24.474975 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold I0619 08:46:24.475332 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold I0619 08:46:24.475668 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold I0619 08:46:24.475931 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold I0619 08:46:24.476265 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold I0619 08:46:24.476573 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold I0619 08:46:24.476926 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold I0619 08:46:24.477203 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold I0619 08:46:24.477541 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold I0619 08:46:24.477811 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold I0619 08:46:24.478186 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold I0619 08:46:24.478434 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold I0619 08:46:24.478747 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold I0619 08:46:24.478998 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold I0619 08:46:24.479341 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold I0619 08:46:24.479565 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold I0619 08:46:24.479938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold I0619 08:46:24.480194 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold I0619 08:46:24.480589 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold I0619 08:46:24.480898 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold I0619 08:46:24.481264 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold I0619 08:46:24.481486 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold I0619 08:46:24.481839 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold I0619 08:46:24.482104 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold I0619 08:46:24.482481 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold I0619 08:46:24.482767 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold I0619 08:46:24.483124 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold I0619 08:46:24.483347 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold I0619 08:46:24.483694 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold I0619 08:46:24.483980 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold I0619 08:46:24.484319 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold I0619 08:46:24.484550 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold I0619 08:46:24.484876 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold I0619 08:46:24.485111 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold I0619 08:46:24.485412 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold I0619 08:46:24.485624 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold I0619 08:46:24.485938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold I0619 08:46:24.486167 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold I0619 08:46:24.486395 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold I0619 08:46:24.486675 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold I0619 08:46:24.486938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold I0619 08:46:24.487241 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold I0619 08:46:24.487478 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold WARNING:tensorflow:From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0619 08:46:24.518594 140679256610688 deprecation.py:323] From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. W0619 08:46:24.766963 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. INFO:tensorflow:Done calling model_fn. I0619 08:46:25.693374 140679256610688 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2020-06-19T08:46:25Z I0619 08:46:25.713920 140679256610688 evaluation.py:255] Starting evaluation at 2020-06-19T08:46:25Z INFO:tensorflow:Graph was finalized. I0619 08:46:26.684044 140679256610688 monitored_session.py:240] Graph was finalized. 2020-06-19 08:46:26.685194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.685930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2020-06-19 08:46:26.686050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:46:26.686083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-06-19 08:46:26.686112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-06-19 08:46:26.686141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-06-19 08:46:26.686169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-06-19 08:46:26.686195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-06-19 08:46:26.686228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:46:26.686329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.687075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.687713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2020-06-19 08:46:26.687792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-06-19 08:46:26.687810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2020-06-19 08:46:26.687819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2020-06-19 08:46:26.687940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.688724: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.689491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from training/model.ckpt-2492 I0619 08:46:26.690679 140679256610688 saver.py:1284] Restoring parameters from training/model.ckpt-2492 INFO:tensorflow:Running local_init_op. I0619 08:46:28.285569 140679256610688 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0619 08:46:28.471102 140679256610688 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 2 images. I0619 08:46:31.930884 140675338995456 coco_evaluation.py:237] Performing evaluation on 2 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0619 08:46:31.931652 140675338995456 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.00s) I0619 08:46:31.932090 140675338995456 coco_tools.py:138] DONE (t=0.00s) creating index... index created! 2020-06-19 08:46:31.949144: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[{{node IteratorGetNext}}]] (1) Out of range: End of sequence [[{{node IteratorGetNext}}]] [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once session.run(eval_ops, feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise raise value File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1418, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1176, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] (1) Out of range: End of sequence [[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'IteratorGetNext': File "content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate self._evaluate_build_graph(input_fn, hooks, checkpoint_path)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph self._call_model_fn_eval(input_fn, self.config)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1544, in _call_model_fn_eval input_fn, ModeKeys.EVAL) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/util.py", line 65, in parse_input_fn_result result = iterator.get_next() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next output_shapes=output_shapes, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[{{node PyFunc_3}}]] [[cond/Detections_Left_Groundtruth_Right/0/_4927]] (1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[{{node PyFunc_3}}]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "/content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise raise value File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 511, in _evaluate output_dir=self.eval_dir(name)) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1619, in _evaluate_run config=self._session_config) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once session.run(eval_ops, feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 861, in __exit__ self._close_internal(exception_type) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 894, in _close_internal h.end(self._coordinated_creator.tf_sess) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 951, in end self._final_ops, feed_dict=self._final_ops_feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] [[cond/Detections_Left_Groundtruth_Right/0/_4927]] (1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'PyFunc_3': File "content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate self._evaluate_build_graph(input_fn, hooks, checkpoint_path)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph self._call_model_fn_eval(input_fn, self.config)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1547, in _call_model_fn_eval features, labels, ModeKeys.EVAL, config) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) File "content/models/research/object_detection/model_lib.py", line 570, in model_fn eval_config, list(category_index.values()), eval_dict) File "content/models/research/object_detection/eval_util.py", line 1045, in get_eval_metric_ops_for_evaluators eval_dict)) File "content/models/research/object_detection/metrics/coco_evaluation.py", line 426, in get_estimator_eval_metric_ops first_value_op = tf.py_func(first_value_func, [], tf.float32) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 513, in py_func return py_func_common(func, inp, Tout, stateful, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 495, in py_func_common func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 318, in _internal_py_func input=inp, token=token, Tout=Tout, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_script_ops.py", line 170, in py_func "PyFunc", input=input, token=token, Tout=Tout, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack()``` What are the possibilites of getting over this issue, any kind of recommendations?
you need numpy version 1.16.x
csingh27
commented
3 years ago I am still struggling with this error. Any solutions ? And no numpy 1.16.x does not solve the issue
Totemi1324
commented
3 years ago No, numpy is not the issue. For me really, experimenting with batch size helped
msevnctkn
commented
3 years ago i still deal with same problem, i havent solved yet. have a solution about it?
msevnctkn
commented
3 years ago @Rajat-Mehta @Totemi1324 @tombstone @jch1 @pkulzc I am training a custom ssd_mobilenet_v2_quantized_300x300 TensorFlow model for object detection using Google Colab with the downgraded version of TensorFlow 1.15.2 because I use to train my model on previous version of TensorFlow i.e. 1.14.0 but due to the latest update to version 2.2.0, I get the strange errors and therefore I can't use the latest version. Using 1.15.2 version and selection even batch size, I successfully starts the training process but after some time, the training process stops with the following errors.
TypeError: 'numpy.float64' object cannot be interpreted as an integerself._traceback = tf_stack.extract_stack()Complete training log can be seen as follows;WARNING:tensorflow:Forced number of epochs for all eval validations to be 1. W0619 08:34:55.917273 140679256610688 model_lib.py:717] Forced number of epochs for all eval validations to be 1. INFO:tensorflow:Maybe overwriting train_steps: 50000 I0619 08:34:55.917532 140679256610688 config_util.py:523] Maybe overwriting train_steps: 50000 INFO:tensorflow:Maybe overwriting use_bfloat16: False I0619 08:34:55.917701 140679256610688 config_util.py:523] Maybe overwriting use_bfloat16: False INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1 I0619 08:34:55.917855 140679256610688 config_util.py:523] Maybe overwriting sample_1_of_n_eval_examples: 1 INFO:tensorflow:Maybe overwriting eval_num_epochs: 1 I0619 08:34:55.917991 140679256610688 config_util.py:523] Maybe overwriting eval_num_epochs: 1 INFO:tensorflow:Maybe overwriting load_pretrained: True I0619 08:34:55.918123 140679256610688 config_util.py:523] Maybe overwriting load_pretrained: True INFO:tensorflow:Ignoring config override key: load_pretrained I0619 08:34:55.918251 140679256610688 config_util.py:533] Ignoring config override key: load_pretrained WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. W0619 08:34:55.919228 140679256610688 model_lib.py:733] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False I0619 08:34:55.919431 140679256610688 model_lib.py:768] create_estimator_and_inputs: use_tpu False, export_to_tpu False INFO:tensorflow:Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} I0619 08:34:55.919958 140679256610688 estimator.py:212] Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff21da66128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator. W0619 08:34:55.920258 140679256610688 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7ff203e51a60>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Not using Distribute Coordinator. I0619 08:34:55.921080 140679256610688 estimator_training.py:186] Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). I0619 08:34:55.921281 140679256610688 training.py:612] Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. I0619 08:34:55.921539 140679256610688 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W0619 08:34:55.935507 140679256610688 deprecation.py:323] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W0619 08:34:55.974960 140679256610688 dataset_builder.py:83] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. W0619 08:34:55.981717 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() W0619 08:34:56.005888 140679256610688 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:175: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' W0619 08:34:56.041087 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff20385a400>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' WARNING:tensorflow:Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 W0619 08:34:56.271473 140679256610688 ag_logging.py:146] Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7ff229279488> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. W0619 08:34:56.278620 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:79: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. WARNING:tensorflow:From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0619 08:34:56.287626 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W0619 08:34:56.404175 140679256610688 deprecation.py:323] From /content/models/research/object_detection/core/preprocessor.py:198: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0619 08:34:57.319203 140679256610688 deprecation.py:323] From /content/models/research/object_detection/inputs.py:260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. INFO:tensorflow:Calling model_fn. I0619 08:34:58.000243 140679256610688 estimator.py:1148] Calling model_fn. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. W0619 08:34:58.086804 140679256610688 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.276282 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.313794 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.349460 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.385352 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.422045 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:35:01.460603 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold I0619 08:35:12.419386 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold I0619 08:35:12.419960 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold I0619 08:35:12.420622 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold I0619 08:35:12.421021 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold I0619 08:35:12.421649 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold I0619 08:35:12.422075 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold I0619 08:35:12.422821 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold I0619 08:35:12.423317 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold I0619 08:35:12.423930 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold I0619 08:35:12.424316 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold I0619 08:35:12.425071 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold I0619 08:35:12.425590 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold I0619 08:35:12.426191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold I0619 08:35:12.426571 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold I0619 08:35:12.427229 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold I0619 08:35:12.427820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold I0619 08:35:12.428410 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold I0619 08:35:12.428805 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold I0619 08:35:12.429387 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold I0619 08:35:12.429798 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold I0619 08:35:12.430403 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold I0619 08:35:12.430883 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold I0619 08:35:12.431473 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold I0619 08:35:12.431915 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold I0619 08:35:12.432562 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold I0619 08:35:12.433037 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold I0619 08:35:12.433677 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold I0619 08:35:12.434067 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold I0619 08:35:12.434737 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold I0619 08:35:12.435177 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold I0619 08:35:12.435865 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold I0619 08:35:12.436269 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold I0619 08:35:12.436948 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold I0619 08:35:12.437350 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold I0619 08:35:12.438012 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold I0619 08:35:12.438425 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold I0619 08:35:12.438820 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold I0619 08:35:12.439191 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold I0619 08:35:12.439666 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold I0619 08:35:12.440045 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold I0619 08:35:12.440409 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold I0619 08:35:12.440825 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold I0619 08:35:12.441197 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0619 08:35:22.078743 140679256610688 deprecation.py:506] From /tensorflow-1.15.2/python3.6/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. I0619 08:35:30.469601 140679256610688 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. I0619 08:35:30.471154 140679256610688 basic_session_run_hooks.py:541] Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. I0619 08:35:37.414344 140679256610688 monitored_session.py:240] Graph was finalized. 2020-06-19 08:35:37.428439: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz 2020-06-19 08:35:37.429078: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-19 08:35:37.429122: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-06-19 08:35:37.434521: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-06-19 08:35:37.613365: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:37.614447: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f7f6e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-06-19 08:35:37.614487: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2020-06-19 08:35:37.615832: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:37.616699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2020-06-19 08:35:37.617137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:35:37.869500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-06-19 08:35:37.990427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-06-19 08:35:38.025559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-06-19 08:35:38.280571: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-06-19 08:35:38.323569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-06-19 08:35:38.816387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:35:38.816668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.817676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.818508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2020-06-19 08:35:38.822059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:35:38.823827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-06-19 08:35:38.823860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2020-06-19 08:35:38.823871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2020-06-19 08:35:38.825211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.826359: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:35:38.827380: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2020-06-19 08:35:38.827455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) INFO:tensorflow:Running local_init_op. I0619 08:35:51.728541 140679256610688 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0619 08:35:52.421987 140679256610688 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into training/model.ckpt. I0619 08:36:11.138946 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 0 into training/model.ckpt. 2020-06-19 08:36:32.110483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:36:36.543281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 INFO:tensorflow:loss = 20.36485, step = 0 I0619 08:36:38.734370 140679256610688 basic_session_run_hooks.py:262] loss = 20.36485, step = 0 INFO:tensorflow:global_step/sec: 2.70978 I0619 08:37:15.636997 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 2.70978 INFO:tensorflow:loss = 2.5266151, step = 100 (36.904 sec) I0619 08:37:15.637996 140679256610688 basic_session_run_hooks.py:260] loss = 2.5266151, step = 100 (36.904 sec) INFO:tensorflow:global_step/sec: 4.4168 I0619 08:37:38.277826 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4168 INFO:tensorflow:loss = 3.0113943, step = 200 (22.641 sec) I0619 08:37:38.278793 140679256610688 basic_session_run_hooks.py:260] loss = 3.0113943, step = 200 (22.641 sec) INFO:tensorflow:global_step/sec: 4.44349 I0619 08:38:00.782637 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44349 INFO:tensorflow:loss = 1.3504767, step = 300 (22.505 sec) I0619 08:38:00.783768 140679256610688 basic_session_run_hooks.py:260] loss = 1.3504767, step = 300 (22.505 sec) INFO:tensorflow:global_step/sec: 4.43355 I0619 08:38:23.337961 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43355 INFO:tensorflow:loss = 2.7894356, step = 400 (22.555 sec) I0619 08:38:23.339154 140679256610688 basic_session_run_hooks.py:260] loss = 2.7894356, step = 400 (22.555 sec) INFO:tensorflow:global_step/sec: 4.43814 I0619 08:38:45.869889 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43814 INFO:tensorflow:loss = 6.171316, step = 500 (22.532 sec) I0619 08:38:45.870794 140679256610688 basic_session_run_hooks.py:260] loss = 6.171316, step = 500 (22.532 sec) INFO:tensorflow:global_step/sec: 4.41364 I0619 08:39:08.526993 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41364 INFO:tensorflow:loss = 2.9929922, step = 600 (22.657 sec) I0619 08:39:08.528079 140679256610688 basic_session_run_hooks.py:260] loss = 2.9929922, step = 600 (22.657 sec) INFO:tensorflow:global_step/sec: 4.44947 I0619 08:39:31.001498 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44947 INFO:tensorflow:loss = 4.0605145, step = 700 (22.474 sec) I0619 08:39:31.002526 140679256610688 basic_session_run_hooks.py:260] loss = 4.0605145, step = 700 (22.474 sec) INFO:tensorflow:global_step/sec: 4.43146 I0619 08:39:53.567426 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43146 INFO:tensorflow:loss = 3.246991, step = 800 (22.568 sec) I0619 08:39:53.570216 140679256610688 basic_session_run_hooks.py:260] loss = 3.246991, step = 800 (22.568 sec) INFO:tensorflow:global_step/sec: 4.43037 I0619 08:40:16.138887 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43037 INFO:tensorflow:loss = 3.6994023, step = 900 (22.570 sec) I0619 08:40:16.139893 140679256610688 basic_session_run_hooks.py:260] loss = 3.6994023, step = 900 (22.570 sec) INFO:tensorflow:global_step/sec: 4.44706 I0619 08:40:38.625646 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44706 INFO:tensorflow:loss = 3.306572, step = 1000 (22.487 sec) I0619 08:40:38.626590 140679256610688 basic_session_run_hooks.py:260] loss = 3.306572, step = 1000 (22.487 sec) INFO:tensorflow:global_step/sec: 4.42377 I0619 08:41:01.230800 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.42377 INFO:tensorflow:loss = 1.8826125, step = 1100 (22.605 sec) I0619 08:41:01.231729 140679256610688 basic_session_run_hooks.py:260] loss = 1.8826125, step = 1100 (22.605 sec) INFO:tensorflow:global_step/sec: 4.439 I0619 08:41:23.758457 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.439 INFO:tensorflow:loss = 2.5373318, step = 1200 (22.528 sec) I0619 08:41:23.759881 140679256610688 basic_session_run_hooks.py:260] loss = 2.5373318, step = 1200 (22.528 sec) INFO:tensorflow:global_step/sec: 4.43686 I0619 08:41:46.296886 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43686 INFO:tensorflow:loss = 3.7123256, step = 1300 (22.538 sec) I0619 08:41:46.297836 140679256610688 basic_session_run_hooks.py:260] loss = 3.7123256, step = 1300 (22.538 sec) INFO:tensorflow:global_step/sec: 4.37992 I0619 08:42:09.128318 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.37992 INFO:tensorflow:loss = 2.2112222, step = 1400 (22.832 sec) I0619 08:42:09.129705 140679256610688 basic_session_run_hooks.py:260] loss = 2.2112222, step = 1400 (22.832 sec) INFO:tensorflow:global_step/sec: 4.44905 I0619 08:42:31.605058 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44905 INFO:tensorflow:loss = 1.3892399, step = 1500 (22.476 sec) I0619 08:42:31.606064 140679256610688 basic_session_run_hooks.py:260] loss = 1.3892399, step = 1500 (22.476 sec) INFO:tensorflow:global_step/sec: 4.45304 I0619 08:42:54.061597 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.45304 INFO:tensorflow:loss = 2.1145034, step = 1600 (22.457 sec) I0619 08:42:54.062722 140679256610688 basic_session_run_hooks.py:260] loss = 2.1145034, step = 1600 (22.457 sec) INFO:tensorflow:global_step/sec: 4.43729 I0619 08:43:16.597952 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43729 INFO:tensorflow:loss = 1.0390191, step = 1700 (22.536 sec) I0619 08:43:16.598960 140679256610688 basic_session_run_hooks.py:260] loss = 1.0390191, step = 1700 (22.536 sec) INFO:tensorflow:global_step/sec: 4.41662 I0619 08:43:39.239702 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41662 INFO:tensorflow:loss = 3.100757, step = 1800 (22.642 sec) I0619 08:43:39.241185 140679256610688 basic_session_run_hooks.py:260] loss = 3.100757, step = 1800 (22.642 sec) INFO:tensorflow:global_step/sec: 4.39809 I0619 08:44:01.976768 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.39809 INFO:tensorflow:loss = 3.6370416, step = 1900 (22.737 sec) I0619 08:44:01.977803 140679256610688 basic_session_run_hooks.py:260] loss = 3.6370416, step = 1900 (22.737 sec) INFO:tensorflow:global_step/sec: 4.43766 I0619 08:44:24.511184 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766 INFO:tensorflow:loss = 2.2865329, step = 2000 (22.535 sec) I0619 08:44:24.512323 140679256610688 basic_session_run_hooks.py:260] loss = 2.2865329, step = 2000 (22.535 sec) INFO:tensorflow:global_step/sec: 4.41886 I0619 08:44:47.141552 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.41886 INFO:tensorflow:loss = 2.2199974, step = 2100 (22.630 sec) I0619 08:44:47.142618 140679256610688 basic_session_run_hooks.py:260] loss = 2.2199974, step = 2100 (22.630 sec) INFO:tensorflow:global_step/sec: 4.44154 I0619 08:45:09.656170 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.44154 INFO:tensorflow:loss = 2.1248786, step = 2200 (22.515 sec) I0619 08:45:09.657206 140679256610688 basic_session_run_hooks.py:260] loss = 2.1248786, step = 2200 (22.515 sec) INFO:tensorflow:global_step/sec: 4.43766 I0619 08:45:32.190551 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.43766 INFO:tensorflow:loss = 4.8454585, step = 2300 (22.535 sec) I0619 08:45:32.191816 140679256610688 basic_session_run_hooks.py:260] loss = 4.8454585, step = 2300 (22.535 sec) INFO:tensorflow:global_step/sec: 4.4216 I0619 08:45:54.806908 140679256610688 basic_session_run_hooks.py:692] global_step/sec: 4.4216 INFO:tensorflow:loss = 0.9450711, step = 2400 (22.616 sec) I0619 08:45:54.808129 140679256610688 basic_session_run_hooks.py:260] loss = 0.9450711, step = 2400 (22.616 sec) INFO:tensorflow:Saving checkpoints for 2492 into training/model.ckpt. I0619 08:46:15.349621 140679256610688 basic_session_run_hooks.py:606] Saving checkpoints for 2492 into training/model.ckpt. WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' W0619 08:46:18.131156 140679256610688 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7ff1ea0d0c18>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num' WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 W0619 08:46:18.347941 140679256610688 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7ff1f92afb70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 INFO:tensorflow:Calling model_fn. I0619 08:46:18.968489 140679256610688 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.590440 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.626965 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.665331 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.702586 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.738639 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0619 08:46:21.774662 140679256610688 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold I0619 08:46:24.474975 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold I0619 08:46:24.475332 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold I0619 08:46:24.475668 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold I0619 08:46:24.475931 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold I0619 08:46:24.476265 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold I0619 08:46:24.476573 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold I0619 08:46:24.476926 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold I0619 08:46:24.477203 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold I0619 08:46:24.477541 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold I0619 08:46:24.477811 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold I0619 08:46:24.478186 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold I0619 08:46:24.478434 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold I0619 08:46:24.478747 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold I0619 08:46:24.478998 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold I0619 08:46:24.479341 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold I0619 08:46:24.479565 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold I0619 08:46:24.479938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold I0619 08:46:24.480194 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold I0619 08:46:24.480589 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold I0619 08:46:24.480898 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold I0619 08:46:24.481264 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold I0619 08:46:24.481486 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold I0619 08:46:24.481839 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold I0619 08:46:24.482104 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold I0619 08:46:24.482481 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold I0619 08:46:24.482767 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold I0619 08:46:24.483124 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold I0619 08:46:24.483347 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold I0619 08:46:24.483694 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold I0619 08:46:24.483980 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold I0619 08:46:24.484319 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold I0619 08:46:24.484550 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold I0619 08:46:24.484876 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/expand/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold I0619 08:46:24.485111 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold I0619 08:46:24.485412 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/Conv_1/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold I0619 08:46:24.485624 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold I0619 08:46:24.485938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold I0619 08:46:24.486167 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold I0619 08:46:24.486395 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold I0619 08:46:24.486675 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold I0619 08:46:24.486938 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold I0619 08:46:24.487241 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/add_fold INFO:tensorflow:Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold I0619 08:46:24.487478 140679256610688 quantize.py:299] Skipping quant after FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/add_fold WARNING:tensorflow:From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0619 08:46:24.518594 140679256610688 deprecation.py:323] From /content/models/research/object_detection/eval_util.py:830: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. W0619 08:46:24.766963 140679256610688 deprecation.py:323] From /content/models/research/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. INFO:tensorflow:Done calling model_fn. I0619 08:46:25.693374 140679256610688 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2020-06-19T08:46:25Z I0619 08:46:25.713920 140679256610688 evaluation.py:255] Starting evaluation at 2020-06-19T08:46:25Z INFO:tensorflow:Graph was finalized. I0619 08:46:26.684044 140679256610688 monitored_session.py:240] Graph was finalized. 2020-06-19 08:46:26.685194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.685930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2020-06-19 08:46:26.686050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2020-06-19 08:46:26.686083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-06-19 08:46:26.686112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2020-06-19 08:46:26.686141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2020-06-19 08:46:26.686169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2020-06-19 08:46:26.686195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2020-06-19 08:46:26.686228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-06-19 08:46:26.686329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.687075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.687713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2020-06-19 08:46:26.687792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-06-19 08:46:26.687810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2020-06-19 08:46:26.687819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2020-06-19 08:46:26.687940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.688724: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-06-19 08:46:26.689491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from training/model.ckpt-2492 I0619 08:46:26.690679 140679256610688 saver.py:1284] Restoring parameters from training/model.ckpt-2492 INFO:tensorflow:Running local_init_op. I0619 08:46:28.285569 140679256610688 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0619 08:46:28.471102 140679256610688 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 2 images. I0619 08:46:31.930884 140675338995456 coco_evaluation.py:237] Performing evaluation on 2 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0619 08:46:31.931652 140675338995456 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.00s) I0619 08:46:31.932090 140675338995456 coco_tools.py:138] DONE (t=0.00s) creating index... index created! 2020-06-19 08:46:31.949144: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[{{node IteratorGetNext}}]] (1) Out of range: End of sequence [[{{node IteratorGetNext}}]] [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once session.run(eval_ops, feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise raise value File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1418, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1176, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] (1) Out of range: End of sequence [[node IteratorGetNext (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5/_4683]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'IteratorGetNext': File "content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate self._evaluate_build_graph(input_fn, hooks, checkpoint_path)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph self._call_model_fn_eval(input_fn, self.config)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1544, in _call_model_fn_eval input_fn, ModeKeys.EVAL) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/util.py", line 65, in parse_input_fn_result result = iterator.get_next() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next output_shapes=output_shapes, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[{{node PyFunc_3}}]] [[cond/Detections_Left_Groundtruth_Right/0/_4927]] (1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[{{node PyFunc_3}}]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "/content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise raise value File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 511, in _evaluate output_dir=self.eval_dir(name)) File "/tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1619, in _evaluate_run config=self._session_config) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once session.run(eval_ops, feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 861, in __exit__ self._close_internal(exception_type) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 894, in _close_internal h.end(self._coordinated_creator.tf_sess) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 951, in end self._final_ops, feed_dict=self._final_ops_feed_dict) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] [[cond/Detections_Left_Groundtruth_Right/0/_4927]] (1) Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 235, in __call__ ret = func(*args) File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 416, in first_value_func self._metrics = self.evaluate() File "/content/models/research/object_detection/metrics/coco_evaluation.py", line 247, in evaluate coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False) File "/content/models/research/object_detection/metrics/coco_tools.py", line 178, in __init__ cocoeval.COCOeval.__init__(self, groundtruth, detections, iouType=iou_type) File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 76, in __init__ self.params = Params(iouType=iouType) # parameters File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 527, in __init__ self.setDetParams() File "/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py", line 507, in setDetParams self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True) File "<__array_function__ internals>", line 6, in linspace File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace .format(type(num))) TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. [[node PyFunc_3 (defined at tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'PyFunc_3': File "content/models/research/object_detection/model_main.py", line 114, in <module> tf.app.run() File "tensorflow-1.15.2/python3.6/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "content/models/research/object_detection/model_main.py", line 110, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate return executor.run() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 613, in run return self.run_local() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 714, in run_local saving_listeners=saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "tensorflow-1.15.2/python3.6/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save if l.after_save(session, step): File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 519, in after_save self._evaluate(global_step_value) # updates self.eval_result File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate self._evaluator.evaluate_and_export()) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export hooks=self._eval_spec.hooks) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate name=name) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval return _evaluate() File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate self._evaluate_build_graph(input_fn, hooks, checkpoint_path)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph self._call_model_fn_eval(input_fn, self.config)) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1547, in _call_model_fn_eval features, labels, ModeKeys.EVAL, config) File "tensorflow-1.15.2/python3.6/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) File "content/models/research/object_detection/model_lib.py", line 570, in model_fn eval_config, list(category_index.values()), eval_dict) File "content/models/research/object_detection/eval_util.py", line 1045, in get_eval_metric_ops_for_evaluators eval_dict)) File "content/models/research/object_detection/metrics/coco_evaluation.py", line 426, in get_estimator_eval_metric_ops first_value_op = tf.py_func(first_value_func, [], tf.float32) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 513, in py_func return py_func_common(func, inp, Tout, stateful, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 495, in py_func_common func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/script_ops.py", line 318, in _internal_py_func input=inp, token=token, Tout=Tout, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_script_ops.py", line 170, in py_func "PyFunc", input=input, token=token, Tout=Tout, name=name) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack()``` What are the possibilites of getting over this issue, any kind of recommendations?you need numpy version 1.16.x
bro, you solved it ?
clh-b
commented
3 years ago 
varunjain3
commented
3 years ago I was stuck at this problem a few days back, changing the tf environment variable before running the same script works for me
$ export TF_FORCE_GPU_ALLOW_GROWTH=true
kmer102
commented
3 years ago https://stackoverflow.com/questions/62466877/self-traceback-tf-stack-extract-stack
this works for me
Having the same issue and despite trying to use add those imports, still gets me nowhere. Unsure if anyone else came across the same issue and successfully got it fixed?
jocelynbaduria
commented
2 years ago I am getting also the same error after training DenseNet.
return target(*args, **kwargs)File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 1431, in concat return gen_array_ops.concat_v2(values=values, axis=axis, name=name) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2 "ConcatV2", values=values, axis=axis, name=name) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 793, in _apply_op_helper op_def=op_def) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3360, in create_op attrs, op_def, compute_device) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3429, in _create_op_internal op_def=op_def) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1751, in init self._traceback = tf_stack.extract_stack()
jocelynbaduria
commented
2 years ago I am getting also the same error after training DenseNet.
return target(*args, **kwargs)File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 1431, in concat return gen_array_ops.concat_v2(values=values, axis=axis, name=name) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2 "ConcatV2", values=values, axis=axis, name=name) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 793, in _apply_op_helper op_def=op_def) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, kwargs) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3360, in create_op attrs, op_def, compute_device) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3429, in _create_op_internal op_def=op_def) File "/opt/anaconda3/envs/Selfie_cpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1751, in init** self._traceback = tf_stack.extract_stack()
I am currently running it in CPU not GPU and I solved it using the correct version of Tensorflow 2.0 and Python 3.7.7.
System information
What is the top-level directory of the model you are using: tensorflow/models
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 64bit
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): gpu-1.15
Have I written custom code: Yes. I added these lines in order to get the training started at all:
config = ConfigProto() config.gpu_options.allow_growth = True session = InteractiveSession(config=config)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()
(0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
INFO:tensorflow:global step 1474: loss = 7.0426 (0.075 sec/step) I0518 12:43:10.650701 12884 learning.py:507] global step 1474: loss = 7.0426 (0.075 sec/step) INFO:tensorflow:global step 1475: loss = 7.8581 (0.076 sec/step) I0518 12:43:10.726234 12884 learning.py:507] global step 1475: loss = 7.8581 (0.076 sec/step) INFO:tensorflow:global step 1476: loss = 10.0487 (0.105 sec/step) I0518 12:43:10.831919 12884 learning.py:507] global step 1476: loss = 10.0487 (0.105 sec/step) INFO:tensorflow:Error reported to Coordinator: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] [[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]] (1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance': File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 355, in train
model_var.op.name, model_var))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(args)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]]
[[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]]
(1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 495, in run self.run_loop() File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\supervisor.py", line 1045, in run_loop [self._sv.summary_op, self._sv.global_step]) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run run_metadata_ptr) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run run_metadata) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] [[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]] (1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance': File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 355, in train
model_var.op.name, model_var))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, **kwargs)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
I0518 12:43:10.883035 11372 coordinator.py:219] Error reported to Coordinator: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] [[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]] (1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance': File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 355, in train
model_var.op.name, model_var))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(args)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]]
[[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]]
(1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 495, in run self.run_loop() File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\supervisor.py", line 1045, in run_loop [self._sv.summary_op, self._sv.global_step]) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run run_metadata_ptr) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run run_metadata) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] [[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]] (1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance': File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 355, in train
model_var.op.name, model_var))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, **kwargs)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
INFO:tensorflow:global step 1477: loss = 9.4083 (0.077 sec/step) I0518 12:43:11.339294 12884 learning.py:507] global step 1477: loss = 9.4083 (0.077 sec/step) INFO:tensorflow:Finished training! Saving model to disk. I0518 12:43:11.352624 12884 learning.py:785] Finished training! Saving model to disk. C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\writer\writer.py:386: UserWarning: Attempting to use a closed FileWriter. The operation will be a noop unless the FileWriter is explicitly reopened. warnings.warn("Attempting to use a closed FileWriter. " Traceback (most recent call last): File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call return fn(*args) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn target_list, run_metadata) File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]] [[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]] (1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance [[{{node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance}}]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 417, in train
saver=saver)
File "C:\tensorflow\lib\site-packages\tensorflow_core\contrib\slim\python\slim\learning.py", line 790, in train
ignore_live_threads=ignore_live_threads)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\supervisor.py", line 839, in stop
ignore_live_threads=ignore_live_threads)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 389, in join
six.reraise(self._exc_info_to_raise)
File "c:\users\nemes\appdata\local\programs\python\python36\lib\site-packages\six.py", line 703, in reraise
raise value
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 297, in stop_on_exception
yield
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\coordinator.py", line 495, in run
self.run_loop()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\training\supervisor.py", line 1045, in run_loop
[self._sv.summary_op, self._sv.global_step])
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
run_metadata_ptr)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
run_metadata)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
[[FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights/read/_63]]
(1) Invalid argument: Nan in summary histogram for: ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance
[[node ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance (defined at C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'ModelVars/FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm/moving_variance': File "object_detection/legacy/train.py", line 191, in
tf.app.run()
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\tensorflow\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
return func(*args, *kwargs)
File "object_detection/legacy/train.py", line 187, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\nemes\Documents\models-master\research\object_detection\legacy\trainer.py", line 355, in train
model_var.op.name, model_var))
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\summary\summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, **kwargs)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
SSD with Mobilenet v2 configuration for MSCOCO Dataset.
Users should configure the fine_tune_checkpoint field in the train config as
well as the label_map_path and input_path fields in the train_input_reader and
eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
should be configured.
model { ssd { num_classes: 20 box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true } } similarity_calculator { iou_similarity { } } anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 } } image_resizer { fixed_shape_resizer { height: 300 width: 300 } } box_predictor { convolutional_box_predictor { min_depth: 0 max_depth: 0 num_layers_before_predictor: 0 use_dropout: false dropout_keep_probability: 0.8 kernel_size: 1 box_code_size: 4 apply_sigmoid_to_scores: false conv_hyperparams { activation: RELU_6, regularizer { l2_regularizer { weight: 0.00004 } } initializer { truncated_normal_initializer { stddev: 0.03 mean: 0.0 } } batch_norm { train: true, scale: true, center: true, decay: 0.9997, epsilon: 0.001, } } } } feature_extractor { type: 'ssd_mobilenet_v2' min_depth: 16 depth_multiplier: 1.0 conv_hyperparams { activation: RELU_6, regularizer { l2_regularizer { weight: 0.00004 } } initializer { truncated_normal_initializer { stddev: 0.03 mean: 0.0 } } batch_norm { train: true, scale: true, center: true, decay: 0.9997, epsilon: 0.001, } } } loss { classification_loss { weighted_sigmoid { } } localization_loss { weighted_smooth_l1 { } } hard_example_miner { num_hard_examples: 3000 iou_threshold: 0.99 loss_type: CLASSIFICATION max_negatives_per_positive: 3 min_negatives_per_image: 3 } classification_weight: 1.0 localization_weight: 1.0 } normalize_loss_by_num_matches: true post_processing { batch_non_max_suppression { score_threshold: 1e-8 iou_threshold: 0.6 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } } }
train_config: { batch_size: 1 optimizer { rms_prop_optimizer: { learning_rate: { exponential_decay_learning_rate { initial_learning_rate: 0.004 decay_steps: 800720 decay_factor: 0.95 } } momentum_optimizer_value: 0.9 decay: 0.9 epsilon: 1.0 } } fine_tune_checkpoint: "C:/Users/nemes/Documents/ssd_mobilenet_v2_coco/model.ckpt" fine_tune_checkpoint_type: "detection"
Note: The below line limits the training process to 200K steps, which we
empirically found to be sufficient enough to train the pets dataset. This
effectively bypasses the learning rate schedule (the learning rate will
never decay). Remove the below line to train indefinitely.
num_steps: 200000 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { ssd_random_crop { } } }
train_input_reader: { tf_record_input_reader { input_path: "C:/Users/nemes/Documents/data/train.record" } label_map_path: "C:/Users/nemes/Documents/data/label_map.pbtxt" }
eval_config: { num_examples: 8000
Note: The below line limits the evaluation process to 10 evaluations.
Remove the below line to evaluate indefinitely.
max_evals: 10 }
eval_input_reader: { tf_record_input_reader { input_path: "C:/Users/nemes/Documents/data/eval.record" } label_map_path: "C:/Users/nemes/Documents/data/label_map.pbtxt" shuffle: false num_readers: 1 }