tensorflow / models

Models and examples built with TensorFlow
Other
77.22k stars 45.75k forks source link

Huge Error When Attempting to Train Using This API #9616

Open DJT777 opened 3 years ago

DJT777 commented 3 years ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

1. The entire URL of the file you are using

http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet101_v1_fpn_640x640_coco17_tpu-8.tar.gz and http://download.tensorflow.org/models/object_detection/tf2/20200711/faster_rcnn_resnet101_v1_640x640_coco17_tpu-8.tar.gz

2. Describe the bug

I am getting this output when training with a customized dataset and modified config file: Streaming output truncated to the last 5000 lines.

       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0.], dtype=float32)>
}, MirroredVariable:{
  0: <tf.Variable 'conv4_block18_3_bn/beta:0' shape=(1024,) dtype=float32, numpy=array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)>
}, MirroredVariable:{
  0: <tf.Variable 'conv3_block4_1_bn/gamma:0' shape=(128,) dtype=float32, numpy=
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)>
}, SyncOnReadVariable:{
  0: <tf.Variable 'conv4_block18_1_bn/moving_variance:0' shape=(256,) dtype=float32, numpy=
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1.], dtype=float32)>
}, MirroredVariable:{
  0: <tf.Variable 'conv2_block3_3_bn/beta:0' shape=(256,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0.], dtype=float32)>
}, SyncOnReadVariable:{
  0: <tf.Variable 'conv4_block19_1_bn/moving_mean:0' shape=(256,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0.], dtype=float32)>
}, SyncOnReadVariable:{
  0: <tf.Variable 'conv4_block10_1_bn/moving_mean:0' shape=(256,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0.], dtype=float32)>
}, MirroredVariable:{

REDACTED FOR LINE SPACE

WARNING:tensorflow:Unresolved object in checkpoint: (root).model
W0106 23:23:31.199386 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._groundtruth_lists
W0106 23:23:31.199696 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._groundtruth_lists
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor
W0106 23:23:31.199785 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor
W0106 23:23:31.199881 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
W0106 23:23:31.199952 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._batched_prediction_tensor_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._box_prediction_head
W0106 23:23:31.200022 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._box_prediction_head

REDACTED FOR LINE SPACE

WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads
W0106 23:23:31.200100 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._sorted_head_names
W0106 23:23:31.200213 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._sorted_head_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers
W0106 23:23:31.200285 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads
W0106 23:23:31.200359 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers
W0106 23:23:31.200425 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._resnet_block_names
W0106 23:23:31.200490 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._resnet_block_names
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers
W0106 23:23:31.200572 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone
W0106 23:23:31.200636 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator
W0106 23:23:31.200699 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._box_prediction_head._box_encoder_layers
W0106 23:23:31.200772 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._box_prediction_head._box_encoder_layers
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background
W0106 23:23:31.200837 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._prediction_heads.class_predictions_with_background
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.0
W0106 23:23:31.200930 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.1
W0106 23:23:31.200996 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.2
W0106 23:23:31.201069 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.3
W0106 23:23:31.201176 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.3
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.4
W0106 23:23:31.201246 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._additional_projection_layers.4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings
W0106 23:23:31.201313 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.class_predictions_with_background
W0106 23:23:31.201396 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.class_predictions_with_background
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers.BoxPredictionTower
W0106 23:23:31.201462 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers.BoxPredictionTower
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers.ClassPredictionTower
W0106 23:23:31.201527 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._head_scope_conv_layers.ClassPredictionTower
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers.0
W0106 23:23:31.201591 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers.0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers.1
W0106 23:23:31.201654 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._coarse_feature_layers.1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-0
W0106 23:23:31.201718 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-1
W0106 23:23:31.201809 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-0
W0106 23:23:31.201875 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-0
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-1
W0106 23:23:31.201987 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-1
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-4
W0106 23:23:31.202074 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-5
W0106 23:23:31.202139 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-5
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-2
W0106 23:23:31.202226 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-2
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-3
W0106 23:23:31.202292 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-3
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-8
W0106 23:23:31.202357 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-8
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-4
W0106 23:23:31.202422 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-4
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-5
W0106 23:23:31.202488 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-5
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-11
W0106 23:23:31.202553 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-11
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-6
W0106 23:23:31.202618 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-6
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-7
W0106 23:23:31.202683 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-7
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-8
W0106 23:23:31.202749 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-8
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-9
W0106 23:23:31.202814 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-9
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-16
W0106 23:23:31.202880 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-16
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-17
W0106 23:23:31.202945 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-17
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-10
W0106 23:23:31.203017 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-10
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-11
W0106 23:23:31.203092 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-11
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-20
W0106 23:23:31.203170 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-20
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-12
W0106 23:23:31.203238 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-12
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-13
W0106 23:23:31.203304 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-13
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-23
W0106 23:23:31.203368 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-23
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-14
W0106 23:23:31.203448 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-14
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-15
W0106 23:23:31.203529 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-15
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-26
W0106 23:23:31.203595 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-26
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-27
W0106 23:23:31.203660 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-27
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-16
W0106 23:23:31.203725 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-16
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-17
W0106 23:23:31.203790 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-17
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-30
W0106 23:23:31.203873 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-30
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-18
W0106 23:23:31.203936 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-18
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-19
W0106 23:23:31.204001 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-19
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-33
W0106 23:23:31.204072 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-33
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-20
W0106 23:23:31.204137 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-20
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-21
W0106 23:23:31.204214 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-21
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-36
W0106 23:23:31.204278 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-36
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-37
W0106 23:23:31.204341 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-37
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-22
W0106 23:23:31.248917 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-22
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-23
W0106 23:23:31.249006 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-23
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-40
W0106 23:23:31.249088 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-40
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-24
W0106 23:23:31.249226 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-24
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-25
W0106 23:23:31.249377 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-25
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-43
W0106 23:23:31.249489 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-43
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-26
W0106 23:23:31.249632 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-26
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-27
W0106 23:23:31.249742 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-27
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-28
W0106 23:23:31.249843 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-28
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-29
W0106 23:23:31.249941 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-29
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-48
W0106 23:23:31.250054 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-48
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-49
W0106 23:23:31.250180 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-49
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-30
W0106 23:23:31.250288 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-30
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-31
W0106 23:23:31.250389 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-31
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-52
W0106 23:23:31.250490 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-52
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-32
W0106 23:23:31.250613 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-32
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-33
W0106 23:23:31.250715 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-33
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-55
W0106 23:23:31.250817 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-55
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-34
W0106 23:23:31.250918 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-34
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-35
W0106 23:23:31.251063 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-35
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-58
W0106 23:23:31.251204 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-58
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-59
W0106 23:23:31.251326 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-59
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-36
W0106 23:23:31.251427 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-36
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-37
W0106 23:23:31.251528 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-37
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-62
W0106 23:23:31.251633 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-62
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-38
W0106 23:23:31.251762 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-38
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-39
W0106 23:23:31.251876 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-39
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-65
W0106 23:23:31.251991 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-65
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-40
W0106 23:23:31.252118 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-40
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-41
W0106 23:23:31.252267 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-41
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-68
W0106 23:23:31.252382 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-68
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-69
W0106 23:23:31.252488 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-69
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-42
W0106 23:23:31.252636 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-42
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-43
W0106 23:23:31.252743 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-43
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-72
W0106 23:23:31.252849 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-72
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-44
W0106 23:23:31.252955 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-44
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-45
W0106 23:23:31.253067 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-45
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-75
W0106 23:23:31.253192 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-75
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-46
W0106 23:23:31.253306 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-46
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-47
W0106 23:23:31.253422 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-47
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-78
W0106 23:23:31.253527 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-78
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-79
W0106 23:23:31.253648 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-79
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-48
W0106 23:23:31.253751 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-48
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-49
W0106 23:23:31.253868 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-49
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-82
W0106 23:23:31.253972 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-82
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-50
W0106 23:23:31.254085 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-50
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-51
W0106 23:23:31.254209 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-51
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-85
W0106 23:23:31.254322 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-85
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-52
W0106 23:23:31.254447 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-52
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-53
W0106 23:23:31.254559 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-53
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-54
W0106 23:23:31.254657 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-54
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-55
W0106 23:23:31.254720 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-55
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-90
W0106 23:23:31.254793 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-90
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-91
W0106 23:23:31.254895 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-91
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-56
W0106 23:23:31.254986 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-56
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-57
W0106 23:23:31.255117 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-57
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-94
W0106 23:23:31.255266 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-94
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-58
W0106 23:23:31.255352 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-58
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-59
W0106 23:23:31.255461 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-59
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-97
W0106 23:23:31.255559 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-97
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-60
W0106 23:23:31.255667 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-60
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-61
W0106 23:23:31.255779 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-61
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-100
W0106 23:23:31.255887 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-100
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-101
W0106 23:23:31.255986 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-101
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-62
W0106 23:23:31.256101 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-62
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-63
W0106 23:23:31.256250 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-63
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-104
W0106 23:23:31.256382 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-104
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-64
W0106 23:23:31.256488 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-64
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-65
W0106 23:23:31.256601 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-65
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-107
W0106 23:23:31.256699 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-107
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-66
W0106 23:23:31.256798 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-66
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-67
W0106 23:23:31.256929 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-67
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-110
W0106 23:23:31.257037 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-110
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-111
W0106 23:23:31.257147 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-111
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-68
W0106 23:23:31.257270 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-68
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-69
W0106 23:23:31.257367 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-69
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-114
W0106 23:23:31.257485 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer-114
(root).model._feature_extractor.classification_backbone.layer_with_weights-155.gamma
W0106 23:23:31.341230 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.beta
W0106 23:23:31.341335 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.moving_mean
W0106 23:23:31.341438 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.moving_variance
W0106 23:23:31.341542 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-155.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-156.kernel
W0106 23:23:31.341648 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-156.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.axis
W0106 23:23:31.341752 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.gamma
W0106 23:23:31.341856 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.beta
W0106 23:23:31.341962 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.moving_mean
W0106 23:23:31.342071 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.moving_variance
W0106 23:23:31.342188 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-157.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-158.kernel
W0106 23:23:31.342311 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-158.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.axis
W0106 23:23:31.342415 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.gamma
W0106 23:23:31.342518 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.beta
W0106 23:23:31.342622 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.moving_mean
W0106 23:23:31.342726 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.moving_variance
W0106 23:23:31.342831 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-159.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-160.kernel
W0106 23:23:31.342934 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-160.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.axis
W0106 23:23:31.343037 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.gamma
W0106 23:23:31.343169 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.beta
W0106 23:23:31.343281 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.moving_mean
W0106 23:23:31.343397 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.moving_variance
W0106 23:23:31.343503 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-161.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-162.kernel
W0106 23:23:31.343608 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-162.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.axis
W0106 23:23:31.343766 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.gamma
W0106 23:23:31.343880 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.beta
W0106 23:23:31.343982 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.moving_mean
W0106 23:23:31.344091 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.moving_variance
W0106 23:23:31.344209 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-163.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-164.kernel
W0106 23:23:31.344312 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-164.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.axis
W0106 23:23:31.344414 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.gamma
W0106 23:23:31.344517 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.beta
W0106 23:23:31.344621 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.moving_mean
W0106 23:23:31.344723 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.moving_variance
W0106 23:23:31.344824 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-165.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-166.kernel
W0106 23:23:31.344925 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-166.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.axis
W0106 23:23:31.345026 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.gamma
W0106 23:23:31.345145 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.beta
W0106 23:23:31.345267 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.moving_mean
W0106 23:23:31.345387 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.moving_variance
W0106 23:23:31.345488 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-167.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-168.kernel
W0106 23:23:31.345637 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-168.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.axis
W0106 23:23:31.345739 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.gamma
W0106 23:23:31.345853 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.beta
W0106 23:23:31.345952 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.moving_mean
W0106 23:23:31.346065 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.moving_variance
W0106 23:23:31.346184 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-169.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-170.kernel
W0106 23:23:31.346288 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-170.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.axis
W0106 23:23:31.346388 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.gamma
W0106 23:23:31.346502 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.beta
W0106 23:23:31.346624 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.moving_mean
W0106 23:23:31.346726 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.moving_variance
W0106 23:23:31.346828 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-171.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-172.kernel
W0106 23:23:31.346928 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-172.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.axis
W0106 23:23:31.347030 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.gamma
W0106 23:23:31.347170 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.beta
W0106 23:23:31.347295 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.moving_mean
W0106 23:23:31.347404 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.moving_variance
W0106 23:23:31.347505 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-173.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-174.kernel
W0106 23:23:31.347647 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-174.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.axis
W0106 23:23:31.347748 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.gamma
W0106 23:23:31.347847 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.beta
W0106 23:23:31.347946 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.moving_mean
W0106 23:23:31.348052 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.moving_variance
W0106 23:23:31.348174 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-175.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-176.kernel
W0106 23:23:31.348282 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-176.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.axis
W0106 23:23:31.348382 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.gamma
W0106 23:23:31.348483 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.beta
W0106 23:23:31.348588 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-177.beta
WARNING:tensorflow:Unresolved object in checkpoint: 
(root).model._feature_extractor.classification_backbone.layer_with_weights-182.kernel
W0106 23:23:31.350454 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-182.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor.classification_backbone.layer_with_weights-183.axis
W0106 23:23:31.350554 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.beta
W0106 23:23:31.378225 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.moving_mean
W0106 23:23:31.378345 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.moving_variance
W0106 23:23:31.378480 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.1.10.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.axis
W0106 23:23:31.378598 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.gamma
W0106 23:23:31.378715 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.beta
W0106 23:23:31.378826 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.moving_mean
W0106 23:23:31.378939 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.moving_variance
W0106 23:23:31.379059 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.1.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.4.axis
W0106 23:23:31.379179 140501308098368 util.py:161] Unresolved object in checkpoint: (r(root).model._box_predictor._base_tower_layers_for_heads.box_encodings.2.7.moving_variance

REDACTED FOR LINESPACE

W0106 23:23:31.397057 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.residual_blocks.1.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.residual_blocks.1.0.bias
W0106 23:23:31.397197 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.residual_blocks.1.0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.0.kernel
W0106 23:23:31.397305 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.axis
W0106 23:23:31.397417 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.gamma
W0106 23:23:31.397588 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.beta
W0106 23:23:31.397691 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.moving_mean
W0106 23:23:31.397791 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.moving_variance
W0106 23:23:31.397891 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.0.1.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.0.kernel
W0106 23:23:31.397995 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.axis
W0106 23:23:31.398103 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.gamma
W0106 23:23:31.398238 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.beta
W0106 23:23:31.398366 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.moving_mean
W0106 23:23:31.398483 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.moving_variance
W0106 23:23:31.398597 140501308098368 util.py:161] Unresolved object in checkpoint: (root).model._feature_extractor._fpn_features_generator.conv_layers.1.1.moving_variance
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0106 23:23:31.398726 140501308098368 util.py:169] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

3. Steps to reproduce

Pipeline Config:

model {
  ssd {
    num_classes: 1
    image_resizer {
      fixed_shape_resizer {
        height: 640
        width: 640
      }
    }
    feature_extractor {
      type: "ssd_resnet101_v1_fpn_keras"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 0.00039999998989515007
          }
        }
        initializer {
          truncated_normal_initializer {
            mean: 0.0
            stddev: 0.029999999329447746
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019165
          scale: true
          epsilon: 0.0010000000474974513
        }
      }
      override_base_feature_extractor_hyperparams: true
      fpn {
        min_level: 3
        max_level: 7
      }
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
        use_matmul_gather: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      weight_shared_convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 0.00039999998989515007
            }
          }
          initializer {
            random_normal_initializer {
              mean: 0.0
              stddev: 0.009999999776482582
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.996999979019165
            scale: true
            epsilon: 0.0010000000474974513
          }
        }
        depth: 256
        num_layers_before_predictor: 4
        kernel_size: 3
        class_prediction_bias_init: -4.599999904632568
      }
    }
    anchor_generator {
      multiscale_anchor_generator {
        min_level: 3
        max_level: 7
        anchor_scale: 4.0
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        scales_per_octave: 2
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 9.99999993922529e-09
        iou_threshold: 0.6000000238418579
        max_detections_per_class: 100
        max_total_detections: 100
        use_static_shapes: false
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid_focal {
          gamma: 2.0
          alpha: 0.25
        }
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    encode_background_as_zeros: true
    normalize_loc_loss_by_codesize: true
    inplace_batchnorm_update: true
    freeze_batchnorm: false
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    random_crop_image {
      min_object_covered: 0.0
      min_aspect_ratio: 0.75
      max_aspect_ratio: 3.0
      min_area: 0.75
      max_area: 1.0
      overlap_thresh: 0.0
    }
  }
  sync_replicas: true
  optimizer {
    momentum_optimizer {
      learning_rate {
        cosine_decay_learning_rate {
          learning_rate_base: 0.03999999910593033
          total_steps: 25000
          warmup_learning_rate: 0.013333000242710114
          warmup_steps: 2000
        }
      }
      momentum_optimizer_value: 0.8999999761581421
    }
    use_moving_average: false
  }
  fine_tune_checkpoint: "/content/ssd_resnet101_v1_fpn_640x640_coco17_tpu-8/checkpoint/ckpt-0"
  num_steps: 25000
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false
  fine_tune_checkpoint_type: "detection"
  use_bfloat16: true
  fine_tune_checkpoint_version: V2
}
train_input_reader {
  label_map_path: "/content/drive/MyDrive/label_map_one_class.pbtxt"
  tf_record_input_reader {
    input_path: "/content/drive/MyDrive/train_one_class.tfrecord"
  }
}
eval_config {
  metrics_set: "coco_detection_metrics"
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "/content/drive/MyDrive/label_map_one_class.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "/content/drive/MyDrive/test_one_class.tfrecord"
  }
}

4. Expected behavior

Train model succesfully. Line 1 of .txt of files : 335 500 001.Black_footed_Albatross Black_Footed_Albatross_0046_18.jpg jpeg 60.0 27.0 325.0 304.0 001 Black_footed_Albatross

H, W, subdir, filename, bbox x, bbox y, bbox width, bbox height, class integer, and class name.

With this script:

def create_tf_example(example):
  # TODO(user): Populate the following variables from your example.
  height = int(example[0]) # Image height
  width = int(example[1]) # Image width
  filename = str(os.path.join(example[2], example[3])) # Filename of the image. Empty if image is not from file
  encoded_filename = str.encode(filename)
  abs_filepath = os.path.join(root_dir, filename)
  open_file = open(abs_filepath, "rb")
  encoded_image_data = open_file.read() # Encoded image bytes
  image_format = str.encode(example[4])

  xmins = [float(example[5])/float(example[1])] # List of normalized left x coordinates in bounding box (1 per box)
  xcalc = float(example[7]) + float(example[5])
  xmaxs = [xcalc/float(example[1])] # List of normalized right x coordinates in bounding box
             # (1 per box)

  ymins = [float(example[6])/float(example[0])] # List of normalized top y coordinates in bounding box (1 per box)
  ycalc = float(example[8])+float(example[6])
  ymaxs = [ycalc/float(example[0])] # List of normalized bottom y coordinates in bounding box
             # (1 per box))

  classes_text = [str.encode("bird")] # List of string class name of bounding box (1 per box)

  classes = [int(1)] # List of integer class id of bounding box (1 per box)

  tf_example = tf.train.Example(features=tf.train.Features(feature={
      'image/height': dataset_util.int64_feature(height),
      'image/width': dataset_util.int64_feature(width),
      'image/filename': dataset_util.bytes_feature(encoded_filename),
      'image/source_id': dataset_util.bytes_feature(encoded_filename),
      'image/encoded': dataset_util.bytes_feature(encoded_image_data),
      'image/format': dataset_util.bytes_feature(image_format),
      'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
      'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
      'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
      'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
      'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
      'image/object/class/label': dataset_util.int64_list_feature(classes),

That script creates my records based on the CUB-2011 dataset.

5. Additional context

I am using this kind of dataset:

TFRecord .txt

335 500 001.Black_footed_Albatross Black_Footed_Albatross_0046_18.jpg jpeg 60.0 27.0 325.0 304.0 001 Black_footed_Albatross

  # TODO(user): Populate the following variables from your example.
  height = int(example[0]) # Image height
  width = int(example[1]) # Image width
  filename = str(os.path.join(example[2], example[3])) # Filename of the image. Empty if image is not from file
  encoded_filename = str.encode(filename)
  abs_filepath = os.path.join(root_dir, filename)
  open_file = open(abs_filepath, "rb")
  encoded_image_data = open_file.read() # Encoded image bytes
  image_format = str.encode(example[4])

  xmins = [float(example[5])/float(example[1])] # List of normalized left x coordinates in bounding box (1 per box)
  xcalc = float(example[7]) + float(example[5])
  xmaxs = [xcalc/float(example[1])] # List of normalized right x coordinates in bounding box
             # (1 per box)

  ymins = [float(example[6])/float(example[0])] # List of normalized top y coordinates in bounding box (1 per box)
  ycalc = float(example[8])+float(example[6])
  ymaxs = [ycalc/float(example[0])] # List of normalized bottom y coordinates in bounding box
             # (1 per box))

  classes_text = [str.encode("bird")] # List of string class name of bounding box (1 per box)

  classes = [int(1)] # List of integer class id of bounding box (1 per box)

  tf_example = tf.train.Example(features=tf.train.Features(feature={
      'image/height': dataset_util.int64_feature(height),
      'image/width': dataset_util.int64_feature(width),
      'image/filename': dataset_util.bytes_feature(encoded_filename),
      'image/source_id': dataset_util.bytes_feature(encoded_filename),
      'image/encoded': dataset_util.bytes_feature(encoded_image_data),
      'image/format': dataset_util.bytes_feature(image_format),
      'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
      'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
      'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
      'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
      'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
      'image/object/class/label': dataset_util.int64_list_feature(classes),

6. System information

Google Colab defaults with this repository's dependencies.

rabbitme commented 3 years ago

I got the same error when I trained CenterNet.

DJT777 commented 3 years ago

@rabbitme

I was able to somehow get these models to train when using a modified version of AbdelRahman's Tf2 API training tutorial scripts. I simply modified his bash file to redirect to my configs and alternate directores and such.

Still does not answer why Tf2's scripts are not training properly. I also recommend placing "PATH TO BE CONFIGURED"/ckpt-0 for some of these detection models. It may be confusing for new people how to configure a model checkpoint in the pipeline when it is just "PATH TO BE CONFIGURED" in the config.

devloper13 commented 3 years ago

@DJT777 I got the same error as wel. Thanks for pointing us to AbdelRahman's code. All I changed was from classifiation to detection and mentioned the checkpoint steps. And it worked.

DJT777 commented 3 years ago

@devloper13 You're welcome!

yieniggu commented 3 years ago

@rabbitme where you able to train the centernet? I'm getting the same error when training a centernet_hg_104. Unfortunately for me, none of @DJT777 or @devloper13 tricks worked for me.

Does any colaborator here have something to say or help with?