SSD MobileNet: Cannot finetune from checkpoint

groakat commented 7 years ago

Hi,

I am trying to finetune SDD MobileNet and I am failing because somehow the variables are not found in the checkpoint even though they are present. Finetunig FasterRCNN on the same dataset works fine btw.

/ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.794763: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/biases not fo
und in checkpoint
2017-07-02 22:59:48.794821: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/BoxEncodingPredictor/biases/
RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.810102: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/biases/
ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.813253: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/moving_variance not found in checkpoint
2017-07-02 22:59:48.813408: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/biases/
RMSProp not found in checkpoint
2017-07-02 22:59:48.813459: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/biases/RMSPro
p_1 not found in checkpoint
2017-07-02 22:59:48.813511: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/weights/Expon
entialMovingAverage not found in checkpoint
2017-07-02 22:59:48.813561: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/biases
not found in checkpoint
2017-07-02 22:59:48.813613: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/weights/RMSPr
op not found in checkpoint
2017-07-02 22:59:48.813666: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/weights/RMSPr
op_1 not found in checkpoint
2017-07-02 22:59:48.813721: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_1/ClassPredictor/weights not f
ound in checkpoint
2017-07-02 22:59:48.818346: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/weights
 not found in checkpoint
2017-07-02 22:59:48.829594: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/weights
/RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.829707: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/biases/Expone
ntialMovingAverage not found in checkpoint
2017-07-02 22:59:48.832935: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/biases/
RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.833103: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/biases not fo
und in checkpoint
2017-07-02 22:59:48.833143: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/weights
/ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.833195: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/moving_mean not found in checkpoint
2017-07-02 22:59:48.833244: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/BoxEncodingPredictor/weights
/RMSProp not found in checkpoint
2017-07-02 22:59:48.833294: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/biases/RMSPro
p not found in checkpoint
2017-07-02 22:59:48.835462: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/biases/RMSPro
p_1 not found in checkpoint
2017-07-02 22:59:48.835508: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/weights not f
ound in checkpoint
2017-07-02 22:59:48.885331: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/weights/RMSPr
op not found in checkpoint
2017-07-02 22:59:48.887146: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/biases/
ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.887275: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/biases/
RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.887440: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/weights/Expon
entialMovingAverage not found in checkpoint
2017-07-02 22:59:48.887446: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/biases
not found in checkpoint
2017-07-02 22:59:48.887461: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/biases/
RMSProp not found in checkpoint
2017-07-02 22:59:48.887467: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_2/ClassPredictor/weights/RMSPr
op_1 not found in checkpoint
2017-07-02 22:59:48.887513: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/gamma/RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.890507: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/weights
 not found in checkpoint
2017-07-02 22:59:48.891118: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/weights
/ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.894821: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/biases/Expone
ntialMovingAverage not found in checkpoint
2017-07-02 22:59:48.899697: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/weights
/RMSProp not found in checkpoint
2017-07-02 22:59:48.902223: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/weights not f
ound in checkpoint
2017-07-02 22:59:48.902278: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/BoxEncodingPredictor/weights
/RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.903933: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/gamma/RMSProp not found in checkpoint
2017-07-02 22:59:48.904057: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/weights/Expon
entialMovingAverage not found in checkpoint
2017-07-02 22:59:48.904106: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/biases not fo
und in checkpoint
2017-07-02 22:59:48.904158: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/biases/RMSPro
p_1 not found in checkpoint
2017-07-02 22:59:48.904214: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/biases/RMSPro
p not found in checkpoint
2017-07-02 22:59:48.904274: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/weights/RMSPr
op not found in checkpoint
2017-07-02 22:59:48.907282: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/biases
not found in checkpoint
2017-07-02 22:59:48.907388: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_3/ClassPredictor/weights/RMSPr
op_1 not found in checkpoint
2017-07-02 22:59:48.955458: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/gamma/ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.957467: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/biases/
RMSProp not found in checkpoint
2017-07-02 22:59:48.957677: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/weights
/ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.957730: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/biases/
ExponentialMovingAverage not found in checkpoint
2017-07-02 22:59:48.957773: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/weights
/RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.957865: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/weights
 not found in checkpoint
2017-07-02 22:59:48.957902: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/weights
/RMSProp not found in checkpoint
2017-07-02 22:59:48.957823: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/BoxEncodingPredictor/biases/
RMSProp_1 not found in checkpoint
2017-07-02 22:59:48.961594: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_point
wise/BatchNorm/gamma not found in checkpoint
2017-07-02 22:59:48.966140: W tensorflow/core/framework/op_kernel.cc:1165] Not found: Key BoxPredictor_4/ClassPredictor/weights/Expon
entialMovingAverage not found in checkpoint

...


Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1285, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1264, in _run_fn
    status, run_metadata)
  File "/opt/conda/lib/python3.6/contextlib.py", line 89, in __exit__
    next(self.gen)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_st
atus
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key BoxPredictor_0/BoxEncodingPredictor/biases/ExponentialMovingAverage not fo
und in checkpoint
         [[Node: save_1/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const
_0_0, save_1/RestoreV2_1/tensor_names, save_1/RestoreV2_1/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/models/object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/tmp/models/object_detection/train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/tmp/models/object_detection/trainer.py", line 290, in train
    saver=saver)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 738, in train
    master, start_standard_services=False, config=session_config) as sess:
  File "/opt/conda/lib/python3.6/contextlib.py", line 82, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/opt/conda/lib/python3.6/site-packages/six.py", line 686, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 953, in managed_session
    start_standard_services=start_standard_services)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 708, in prepare_or_wait_for_session
    init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 273, in prepare_session
    config=config)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 205, in _restore_checkpoint
    saver.restore(sess, ckpt.model_checkpoint_path)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1555, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 896, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1279, in _do_run
    options, run_metadata)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1298, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key BoxPredictor_0/BoxEncodingPredictor/biases/ExponentialMovingAverage not fo
und in checkpoint
         [[Node: save_1/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const
_0_0, save_1/RestoreV2_1/tensor_names, save_1/RestoreV2_1/shape_and_slices)]]

Caused by op 'save_1/RestoreV2_1', defined at:
  File "/tmp/models/object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/tmp/models/object_detection/train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/tmp/models/object_detection/trainer.py", line 275, in train
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 661, in restore_v2
    dtypes=dtypes, name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2534, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1203, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key BoxPredictor_0/BoxEncodingPredictor/biases/ExponentialMovingAverage not found in checkpo
int
         [[Node: save_1/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const
_0_0, save_1/RestoreV2_1/tensor_names, save_1/RestoreV2_1/shape_and_slices)]]

ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/Gather:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
['File "/tmp/models/object_detection/train.py", line 198, in <module>\n    tf.app.run()', 'File "/opt/conda/lib/python3.6/site-packag
es/tensorflow/python/platform/app.py", line 48, in run\n    _sys.exit(main(_sys.argv[:1] + flags_passthrough))', 'File "/tmp/models/o
bject_detection/train.py", line 194, in main\n    worker_job_name, is_chief, FLAGS.train_dir)', 'File "/tmp/models/object_detection/t
rainer.py", line 290, in train\n    saver=saver)', 'File "/opt/conda/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/
learning.py", line 663, in train\n    ready_op = tf_variables.report_uninitialized_variables()', 'File "/opt/conda/lib/python3.6/site
-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', '
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 139, in _add_should_use_warning\n    wrap
ped = TFShouldUseWarningWrapper(x)', 'File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 96,
 in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']

If I inspect the checkpoint, all these variables seem to exist:

FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/gamma (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/moving_mean (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/BatchNorm/moving_variance (DT_FLOAT) [256]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/depthwise_weights (DT_FLOAT) [3,3,256,1]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/depthwise_weights/ExponentialMovingAverage (DT_FLOAT) [3,3,256,1]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/depthwise_weights/RMSProp (DT_FLOAT) [3,3,256,1]
FeatureExtractor/MobilenetV1/Conv2d_6_depthwise/depthwise_weights/RMSProp_1 (DT_FLOAT) [3,3,256,1]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/weights (DT_FLOAT) [1,1,256,512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/weights/ExponentialMovingAverage (DT_FLOAT) [1,1,256,512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/weights/RMSProp (DT_FLOAT) [1,1,256,512]
FeatureExtractor/MobilenetV1/Conv2d_6_pointwise/weights/RMSProp_1 (DT_FLOAT) [1,1,256,512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/depthwise_weights (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/depthwise_weights/ExponentialMovingAverage (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/depthwise_weights/RMSProp (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_7_depthwise/depthwise_weights/RMSProp_1 (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/weights (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/weights/ExponentialMovingAverage (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/weights/RMSProp (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_7_pointwise/weights/RMSProp_1 (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/depthwise_weights (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/depthwise_weights/ExponentialMovingAverage (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/depthwise_weights/RMSProp (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_8_depthwise/depthwise_weights/RMSProp_1 (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/weights (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/weights/ExponentialMovingAverage (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/weights/RMSProp (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_8_pointwise/weights/RMSProp_1 (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/depthwise_weights (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/depthwise_weights/ExponentialMovingAverage (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/depthwise_weights/RMSProp (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_9_depthwise/depthwise_weights/RMSProp_1 (DT_FLOAT) [3,3,512,1]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/beta (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/beta/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/beta/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/beta/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/gamma (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/gamma/ExponentialMovingAverage (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/gamma/RMSProp (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/gamma/RMSProp_1 (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/moving_mean (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/BatchNorm/moving_variance (DT_FLOAT) [512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/ExponentialMovingAverage (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp (DT_FLOAT) [1,1,512,512]
FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp_1 (DT_FLOAT) [1,1,512,512]
global_step (DT_INT64) []

I am using the checkpoint ssd_mobilenet_v1_coco_11_06_2017.tar.gz from the model zoo, tensorflow 1.2 and the master branch of tensorflow/models

xhzcyc commented 7 years ago

How did you train the net? Could you tell me? Thanks

groakat commented 7 years ago

I downloaded the pretraine model from http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_11_06_2017.tar.gz

and followed the tutorial https://github.com/tensorflow/models/blob/master/object_detection/g3doc/running_locally.md using the Oxford Pets sample config file (only changed the number of classes and the paths). If I do the same steps with the pretrained FasterRCNN model, everything works.

groakat commented 7 years ago

BTW Im using Python 3.6, so I had the change an iteritems() to items(), to get rid of this error I am getting from the plain master code

Traceback (most recent call last):
  File "/tmp/models/object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/tmp/models/object_detection/train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/tmp/models/object_detection/trainer.py", line 184, in train
    data_augmentation_options)
  File "/tmp/models/object_detection/trainer.py", line 77, in _create_input_queue
    prefetch_queue_capacity=prefetch_queue_capacity)
  File "/tmp/models/object_detection/core/batcher.py", line 87, in __init__
    for key, tensor in tensor_dict.iteritems()})
AttributeError: 'dict' object has no attribute 'iteritems'

diff --git a/object_detection/core/batcher.py b/object_detection/core/batcher.py
index 2325a5e..c5dfb71 100644
--- a/object_detection/core/batcher.py
+++ b/object_detection/core/batcher.py
@@ -84,7 +84,7 @@ class BatchQueue(object):
     # Remember runtime shapes to unpad tensors after batching.
     runtime_shapes = collections.OrderedDict(
         {(key + rt_shape_str): tf.shape(tensor)
-         for key, tensor in tensor_dict.iteritems()})
+         for key, tensor in tensor_dict.items()})

     all_tensors = tensor_dict
     all_tensors.update(runtime_shapes)

is03wlei commented 7 years ago

I also have similar issue.

python object_detection/train.py --logtostderr --pipeline_config_path=./object_detection/samples/configs/rfcn_resnet101_pets.config --train_dir=object_detection/pet_train

W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_6/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_6/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_6/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_6/bottleneck_v1/conv3/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_7/bottleneck_v1/conv3/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv3/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_8/bottleneck_v1/conv2/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv2/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_9/bottleneck_v1/conv3/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/class_predictions/biases/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/class_predictions/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/class_predictions/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/reduce_depth/biases not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/reduce_depth/biases/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/class_predictions/biases not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/refined_locations/biases/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/reduce_depth/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/reduce_depth/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/refined_locations/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/refined_locations/biases not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageBoxPredictor/refined_locations/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/weights/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta/Momentum not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/weights not found in checkpoint W tensorflow/core/framework/op_kernel.cc:993] Not found: Key SecondStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, Key FirstStageFeatureExtractor/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint [[Node: save_1/RestoreV2_15 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_15/tensor_names, save_1/RestoreV2_15/shape_and_slices)]]

Caused by op u'save_1/RestoreV2_15', defined at: File "object_detection/train.py", line 198, in tf.app.run() File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "object_detection/train.py", line 194, in main worker_job_name, is_chief, FLAGS.train_dir) File "build/bdist.linux-x86_64/egg/object_detection/trainer.py", line 275, in train keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1040, in init self.build() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build restore_sequentially=self._restore_sequentially) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 675, in build restore_sequentially, reshape) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 242, in restore_op [spec.tensor.dtype])[0]) File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2 dtypes=dtypes, name=name) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key FirstStageFeatureExtractor/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint [[Node: save_1/RestoreV2_15 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_15/tensor_names, save_1/RestoreV2_15/shape_and_slices)]]

Traceback (most recent call last): File "object_detection/train.py", line 198, in tf.app.run() File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "object_detection/train.py", line 194, in main worker_job_name, is_chief, FLAGS.train_dir) File "build/bdist.linux-x86_64/egg/object_detection/trainer.py", line 290, in train saver=saver) File "/usr/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 776, in train master, start_standard_services=False, config=session_config) as sess: File "/usr/lib64/python2.7/contextlib.py", line 17, in enter return self.gen.next() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 960, in managed_session self.stop(close_summary_writer=close_summary_writer) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 788, in stop stop_grace_period_secs=self._stop_grace_secs) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 386, in join six.reraise(*self._exc_info_to_raise) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 949, in managed_session start_standard_services=start_standard_services) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 706, in prepare_or_wait_for_session init_feed_dict=self._init_feed_dict, init_fn=self._init_fn) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 256, in prepare_session config=config) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 188, in _restore_checkpoint saver.restore(sess, ckpt.model_checkpoint_path) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1428, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run run_metadata_ptr) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run feed_dict_string, options, run_metadata) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run target_list, options, run_metadata) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key FirstStageFeatureExtractor/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint [[Node: save_1/RestoreV2_15 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_15/tensor_names, save_1/RestoreV2_15/shape_and_slices)]]

Caused by op u'save_1/RestoreV2_15', defined at: File "object_detection/train.py", line 198, in tf.app.run() File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "object_detection/train.py", line 194, in main worker_job_name, is_chief, FLAGS.train_dir) File "build/bdist.linux-x86_64/egg/object_detection/trainer.py", line 275, in train keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1040, in init self.build() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build restore_sequentially=self._restore_sequentially) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 675, in build restore_sequentially, reshape) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 242, in restore_op [spec.tensor.dtype])[0]) File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2 dtypes=dtypes, name=name) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key FirstStageFeatureExtractor/resnet_v1_101/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Momentum not found in checkpoint [[Node: save_1/RestoreV2_15 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_15/tensor_names, save_1/RestoreV2_15/shape_and_slices)]]

chenyuZha commented 7 years ago

@groakat @is03wlei Make sure that the path --train_dir is initialized when you did your training. I had the same problem before, it was because in my --train_dir there was an checkpoint file of a different model, so when you run with the current model, this wrong checkpoint file will be searched and used instead .

is03wlei commented 7 years ago

@chenyuZha How to initialize --train_dir?

chenyuZha commented 7 years ago

@is03wlei just remove the old checkpoints in the folder or you indicate a new path of --train_dir.

is03wlei commented 7 years ago

@chenyuZha It works. Thanks

fib5813 commented 7 years ago

I downloaded the ssd mobilenet tar.gz . I have an empty train_dir, and I have put the checkpoints in the data folder, it still gives me the following error: Traceback (most recent call last): File "train.py", line 198, in <module> tf.app.run() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "train.py", line 194, in main worker_job_name, is_chief, FLAGS.train_dir) File "/usr/local/lib/python3.5/dist-packages/tensorflow/models/object_detection/trainer.py", line 216, in train from_detection_checkpoint=train_config.from_detection_checkpoint) File "/usr/local/lib/python3.5/dist-packages/tensorflow/models/object_detection/meta_architectures/ssd_meta_arch.py", line 590, in restore_fn saver = tf.train.Saver(variables_to_restore) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1139, in __init__ self.build() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1161, in build raise ValueError("No variables to save") ValueError: No variables to save

sreeragh-ar commented 7 years ago

https://stackoverflow.com/questions/45325675/unable-to-load-finetune-checkpoint-in-object-detection-training/45363642#45363642

mathCrazyy commented 6 years ago

hello,i also test the model,ssd_mobilenet_v1_coco_11_06_2017.tar.gz, but it can not rechieve the real-time performance ,isn`t it?

gavincmartin commented 6 years ago

@chenyuZha can you explain what you mean by initializing the train_dir? I am using the checkpoint files provided from ssd_mobilenet_v1_coco_11_06_2017.tar.gz in the model detection zoo, and my pipeline configuration file correctly points to the location of these checkpoint files, yet it still fails to find all of these feature extractors.

ssk1991 commented 6 years ago

@gavincmartin

Your train script is probably something like:

python3 train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/fasterrcnn_inception_coco.config

Do the following:

In your models/research/object-detection directory, create a new empty folder, and call it "train"
Modify your train script to replace the train_dir=training/ with train_dir=train/

Your final train script will be: python3 train.py --logtostderr --train_dir=train/ --pipeline_config_path=training/fasterrcnn_inception_coco.config

Yes, it is absolutely bizarre that this solution works.

WillLiGitHub commented 6 years ago

@chenyuZha I finetune mobilenet for classification and got four files, checkpoint, out_graph-2000.data-00000-of-00001, out_graph-2000.index, out_graph-2000.meta. so which one should delete.

chenyuZha commented 6 years ago

@gavincmartin That means train_dir should be empty.

chenyuZha commented 6 years ago

@WillLiGitHub The checkpoint file contains the paths of all of ckpt that you have trained (in your case you have only one because it seems like you trained just for one ckpt). The rest 3 files is used to export your graph .pb. So if you delete any of 3 you will not success to have .pb file.

WillLiGitHub commented 6 years ago

@chenyuZha
Thanks!! I have find the error, when i saved model variables, I just saved variables_trainable, actualy should saved variables_to_restore. Thanks again!

walkerlala commented 6 years ago

@chenyuZha remove the train dir does work. But this same error occur when restoring from parameters from the same models. Just run it, stop, and run it again without changing anything, and you will see the same error showing up. It is potentially a bug in the implementation.

willSapgreen commented 6 years ago

Similar warning:

WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/BatchNorm/beta/ExponentialMovingAverage] is not available in checkpoint ( just one of multiple warning messages )

happens in ssd_inception_v2_coco_2017_11_17.

zhiaozhou commented 6 years ago

Happens in Faster RCNN with Inception ResNet V2 as well and initiating the train log folder does not change anything

ccthien commented 6 years ago

WARNING:root:Variable [FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta/ExponentialMovingAverage] is not available in checkpoint WARNING:root:Variable [FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta/RMSProp] is not available in checkpoint WARNING:root:Variable [FeatureExtractor/MobilenetV2/Conv/BatchNorm/beta/RMSProp_1] is not available in checkpoint WARNING:root:Variable [FeatureExtractor/MobilenetV2/Conv/BatchNorm/gamma/ExponentialMovingAverage] is not available in checkpoint ... for ssd_mobilenet_v2_coco_2018_03_29

Also faster_rcnn_resnet101_coco_2018_01_28 got the same issue...

varun19299 commented 6 years ago

This issue arises again in the latest commit. ea6d6aa

I think the fix towards this has more to undoing the block changes caused by commit 93b8168.

A manual fix for this works with no other implications (tested with faster_rcnn_inception_resnet_v2, loading from oid checkpoints).

ccthien commented 6 years ago

Thank you Varun, I wonder if reverse commit 93b8168 cause any problem with current version? It do not raise error and warning, but I am not sure if it could make anything wrong on training process...

varun19299 commented 6 years ago

Update

The warning seem to be indicatory of Momentum optimiser parameters, which aren't included in the model zoo checkpoints (I need to verify this, would be grateful if someone does this too).

The checkpoints do contain the weights of layers (as can be seen by printing or logging the graph operators).

Ergo, the warnings are correct, and as @pkulzc suggested, would be deprecated soon (new interfaces expected I guess).

but I am not sure if it could make anything wrong on training process...

I am training a few models (Faster RCNN, SSD, RFCN) with and without this commit and would be able to comment on this, say, tomorrow only.

You can follow a similar thread on the commit too: 93b8168

ccthien commented 6 years ago

I have just tested:

Reverse commit 93b8168 on current master version (19/05/2018 - f798e4b5504b0b7ed08f7b7a03fc5a79f00b9f21) work as expected on faster_rcnn_inception_v2_coco_2018_01_28

ccthien commented 6 years ago

Updated: No need to reverse commit 93b8168, although the warning come, it still work as expected. @pkulzc was right. Tested on version 81d776695821337d3a42b22e92651762b5775d4e model faster_rcnn_inception_v2_coco_2018_01_28.

satendra929 commented 6 years ago

@ccthien Although training has started successfully, the warning still appears. Will the that be a problem ?

ccthien commented 6 years ago

@satendra929 It work without any problem. As @pkulzc said: it is expected behavior, you can check his comment on 93b8168.

tarunluthra commented 6 years ago

@ccthien I tried that 81d7766 commit. But I am getting the same error. Could you please help. NotFoundError (see above for traceback): Key FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta not found in checkpoint [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

ccthien commented 6 years ago

Hi @tarunluthra this should be warnings only, not error. If you get error, that mean you do something else not correctly.

zubairahmed-ai commented 5 years ago

NotFoundError (see above for traceback): Key BoxPredictor_0/BoxEncodingPredictor/biases/ExponentialMovingAverage not found in checkpoint [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

This is the error I'm getting on SSDMobileNetv2 and I have no idea how to fix this, anyone?

zubairahmed-ai commented 5 years ago

I am facing the same issue, does anyone know how to fix this? https://github.com/tensorflow/models/issues/5792

varun19299 commented 5 years ago

Are you fine-tuning from a checkpoint with a different optimiser? For instance if you trained 500 steps with Adam then changed to SGD.

You'll have to not load the moving averages in that case.

On Wed 21 Nov, 2018, 9:40 AM Zubair Ahmed, notifications@github.com wrote:

I am facing the same issue, does anyone know how to fix this? #5792 https://github.com/tensorflow/models/issues/5792

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tensorflow/models/issues/1836#issuecomment-440523113, or mute the thread https://github.com/notifications/unsubscribe-auth/AUME5_WsRTFciyMtcnqaKj0bykXd32rOks5uxNJQgaJpZM4OLuA6 .

--

Thank you, Varun.

zubairahmed-ai commented 5 years ago

Hi Varun

I have not changed the optimiser in the config I am using TFOD API with the standard config taken from model configs

varun19299 commented 5 years ago

Could you post all relevant details regarding this error for reproducibility? Further I don't think this is the same issue as the one being discussed in this thread. Could you open a new issue thread?

On Wed, 21 Nov 2018 at 10:11 AM, Zubair Ahmed notifications@github.com wrote:

Hi Varun

I have not changed the optimiser in the config I am using TFOD API with the standard config taken from model configs

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tensorflow/models/issues/1836#issuecomment-440528027, or mute the thread https://github.com/notifications/unsubscribe-auth/AUME50m2Ho8q1DcfwZAR-IVzA4HyakWeks5uxNmMgaJpZM4OLuA6 .

-- Thanks, Varun.

GopeshAgrawal03 commented 5 years ago

WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From C:\models-master\research\object_detection\builders\dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.experimental.parallel_interleave(...). WARNING:tensorflow:From C:\Python36\lib\site-packages\tensorflow\python\ops\sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead. WARNING:tensorflow:From C:\models-master\research\object_detection\core\preprocessor.py:1218: calling squeeze (from tensorflow.python.ops.array_ops) with squeeze_dims is deprecated and will be removed in a future version. Instructions for updating: Use the axis argument instead WARNING:tensorflow:From C:\models-master\research\object_detection\builders\dataset_builder.py:148: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.batch(..., drop_remainder=True). WARNING:root:Variable [BoxPredictor_0/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[273]], model variable shape: [[18]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_0/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 512, 273]], model variable shape: [[1, 1, 512, 18]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_1/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_1/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 1024, 546]], model variable shape: [[1, 1, 1024, 36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_2/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_2/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 512, 546]], model variable shape: [[1, 1, 512, 36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_3/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_3/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 256, 546]], model variable shape: [[1, 1, 256, 36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_4/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_4/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 256, 546]], model variable shape: [[1, 1, 256, 36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_5/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [BoxPredictor_5/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 128, 546]], model variable shape: [[1, 1, 128, 36]]. This variable will not be initialized from the checkpoint. WARNING:root:Variable [global_step] is not available in checkpoint 2019-03-13 12:27:36.418443: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2019-03-13 12:28:31.962714: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:31.964958: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:31.969555: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:31.978090: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:31.984174: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:31.993207: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.012007: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.025045: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.064996: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.047463: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.053941: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.036923: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.083707: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.085259: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.093388: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.103481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.114972: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.150835: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.166238: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.167507: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.168531: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.198749: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.206182: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.218893: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.226387: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.239455: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.251380: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.290722: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.282422: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.284502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.267633: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.316920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.348611: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.352346: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.359779: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.363853: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.371631: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.381081: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.390489: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.403538: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.414511: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.438108: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.446517: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.458236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.459337: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.469421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.480737: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.488306: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.540084: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.541056: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.548609: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.558241: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.570180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.583434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.589711: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.599259: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.607303: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.616699: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.662235: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.663800: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.669798: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.684057: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.691148: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.719214: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.721101: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.749842: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.753415: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.764854: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.771788: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.782605: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.789833: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.799672: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.808519: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.821661: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.850650: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.860447: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.874550: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.885028: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.900399: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.905645: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.917264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.941463: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.948368: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.956327: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.966819: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.977605: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:32.989652: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.011760: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.016252: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.055726: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.064871: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.078132: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.081974: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.090025: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.100590: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.116510: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.123479: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.133019: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.140865: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.151858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.166796: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.173619: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.181721: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.190831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.237270: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.265762: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.271032: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.278080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.282771: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.290744: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.296915: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.303301: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.316513: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.322389: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.332178: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.339953: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.367566: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.373645: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.380845: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.387404: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.416928: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.426989: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.448957: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.459816: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.462181: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.482560: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.490534: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.500993: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.515840: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.521913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.527655: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.541721: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.547764: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.554975: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.564260: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.574017: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.600617: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.616070: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.634684: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.635444: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.647211: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.653765: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.665583: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.672610: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.703028: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.728574: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.730432: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.734143: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.741764: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.753729: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.764223: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.773380: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.802121: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.809407: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.819209: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.833125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.839351: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.850419: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.858315: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.871483: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.904057: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.906849: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.933860: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.945848: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.952120: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.957523: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.967085: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:33.985915: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.006358: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.010318: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.032319: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.039903: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.051175: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.065236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.072089: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.084684: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.095827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.122220: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.127516: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.150759: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.166424: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.175973: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.183781: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.210134: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.222257: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.233723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.283123: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.283617: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.291620: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.305566: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.315965: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.335197: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.348457: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.354309: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.364451: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.389758: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.391327: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.401724: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.415489: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.423187: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.434863: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.462990: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.484325: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.484910: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.493075: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.502996: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.513968: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.523604: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.549788: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.556792: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.568486: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.580733: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.584599: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.598624: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.604953: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.614114: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.651033: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.655609: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.665262: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.674383: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.681539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.688948: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.697924: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.705312: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.731196: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.736387: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.752168: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.756181: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.763065: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.770543: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.780671: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.788041: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.817503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.834692: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.837968: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.857260: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.872257: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.884723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.888661: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.912081: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.918918: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.931527: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.944633: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.954220: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.962280: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.968968: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.994800: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:34.996336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.010407: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.022368: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.050597: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.059891: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.060996: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.070576: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.084333: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.102868: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.123046: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.132615: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.139478: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.154678: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.188703: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.213123: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.217903: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.224252: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.234773: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.239692: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.266125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.270345: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.279486: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.286194: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.293951: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.313316: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.319763: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.330361: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.339622: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.348825: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.354844: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.381858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.387743: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.391477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.439442: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.452238: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.457814: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.478308: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.479738: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.488745: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.517634: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.530013: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_3_hash_table_1/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.534212: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.541410: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.550487: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.601789: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.586093: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.598732: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_0_hash_table_3/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.558080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_1_hash_table/class tensorflow::lookup::LookupInterface does not exist. 2019-03-13 12:28:35.608392: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at lookup_table_op.cc:653 : Not found: Resource localhost/_2_hash_table_2/class tensorflow::lookup::LookupInterface does not exist. Traceback (most recent call last): File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call return fn(*args) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: image_size must contain 3 elements[4] [[{{node cond_6/RandomCropImage/sample_distorted_bounding_box/SampleDistortedBoundingBoxV2}} = SampleDistortedBoundingBoxV2[T=DT_INT32, area_range=[0.1, 1], aspect_ratio_range=[0.5, 2], max_attempts=100, seed=0, seed2=0, use_image_if_no_bounding_boxes=true, _device="/device:CPU:0"](cond_6/RandomCropImage/Shape, cond_6/RandomCropImage/ExpandDims, cond_6/RandomCropImage/PruneNonOverlappingBoxes/Const)]] [[{{node IteratorGetNext}} = IteratorGetNextoutput_shapes=[[12], [12,300,300,3], [12,2], [12,3], [12,100], [12,100,4], [12,100,5], [12,100,5], [12,100], [12,100], [12,100], [12]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "model_main.py", line 109, in tf.app.run() File "C:\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run _sys.exit(main(argv)) File "model_main.py", line 105, in main tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0]) File "C:\Python36\lib\site-packages\tensorflow\python\estimator\training.py", line 471, in train_and_evaluate return executor.run() File "C:\Python36\lib\site-packages\tensorflow\python\estimator\training.py", line 610, in run return self.run_local() File "C:\Python36\lib\site-packages\tensorflow\python\estimator\training.py", line 711, in run_local saving_listeners=saving_listeners) File "C:\Python36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "C:\Python36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "C:\Python36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1241, in _train_model_default saving_listeners) File "C:\Python36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1471, in _train_with_estimatorspec , loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 671, in run run_metadata=run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1156, in run run_metadata=run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run raise six.reraise(original_exc_info) File "C:\Python36\lib\site-packages\six.py", line 693, in reraise raise value File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1240, in run return self._sess.run(args, *kwargs) File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1312, in run run_metadata=run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1076, in run return self._sess.run(args, **kwargs) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run run_metadata_ptr) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run run_metadata) File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: image_size must contain 3 elements[4] [[{{node cond_6/RandomCropImage/sample_distorted_bounding_box/SampleDistortedBoundingBoxV2}} = SampleDistortedBoundingBoxV2[T=DT_INT32, area_range=[0.1, 1], aspect_ratio_range=[0.5, 2], max_attempts=100, seed=0, seed2=0, use_image_if_no_bounding_boxes=true, _device="/device:CPU:0"](cond_6/RandomCropImage/Shape, cond_6/RandomCropImage/ExpandDims, cond_6/RandomCropImage/PruneNonOverlappingBoxes/Const)]] [[node IteratorGetNext (defined at model_main.py:105) = IteratorGetNextoutput_shapes=[[12], [12,300,300,3], [12,2], [12,3], [12,100], [12,100,4], [12,100,5], [12,100,5], [12,100], [12,100], [12,100], [12]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

GopeshAgrawal03 commented 5 years ago

how could one resolve the above error please help

iriyagupta commented 5 years ago

@groakat @is03wlei Make sure that the path --train_dir is initialized when you did your training. I had the same problem before, it was because in my --train_dir there was an checkpoint file of a different model, so when you run with the current model, this wrong checkpoint file will be searched and used instead .

Hi, Following the same step also results in the same warnings where few of the features aren't available in checkpoint. How can it be resolved for ssdlitemobilenet_coco_v1 ? tensorflow directory is set to 1.9

tensorflowbutler commented 4 years ago

Hi There, We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing. If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

kishoreudata commented 3 years ago

I have three files in checkpoint folder of efficientdet_d1_coco17_tpu-32 model and now I need to give path to fine_tune_checkpoint in pipeline.config Those files are :

checkpoint
cpkt-0.DATA
cpkt-0.index file which one should I give in path and for 1st file which is of file type what will be the file extension ?

tensorflow / models

SSD MobileNet: Cannot finetune from checkpoint #1836