Couldn't restore attention_ocr variables from .index and .data checkpoints

HuichuanLiu commented 7 years ago

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

It must be a bug or a feature request.
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS Sierra 10.12.5
TensorFlow installed from (source or binary): created environment in conda, then installed tf via pip
TensorFlow version (use command below): ('v1.2.0-rc2-21-g12f033d', '1.2.0')
Python version: 2.7
Bazel version (if compiling from source): Not installed
CUDA/cuDNN version: No GPU supported
GPU model and memory: No
Exact command to reproduce: python test.py You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

Hi, I am trying to use attention_ocr in my own data, a simple test is firstly implemented. according to the instructions from How to use a pre-trained model but somehow failed in restoring the checkpoints without explicit error info.

The following condition has been checked:

checkpoint files are complete
right path to the checkpoint
graphs have been imported from .meta
Nothing changes after run saver.train.restore() (predictions remained the same)
No error or hints provided

The checkpoint was downloaded as suggested:

wget http://download.tensorflow.org/models/attention_ocr_2017_05_17.tar.gz
tar xf attention_ocr_2017_05_17.tar.gz
cd attention_ocr_2017_05_17
ls -lh

total 64216
-rw-r-----  1 liuhuichuan  staff    14M  5 18 04:07 model.ckpt-399731.data-00000-of-00001
-rw-r-----  1 liuhuichuan  staff   8.2K  5 18 04:07 model.ckpt-399731.index
-rw-r-----  1 liuhuichuan  staff    17M  5 18 04:07 model.ckpt-399731.meta

The graphs were successfully imported from .meta, but somehow saver couldn't recognize .index and .data files:

print os.path.exists('../attention_ocr_2017_05_17/model.ckpt-399731.data-00000-of-00001')
print os.path.exists('../attention_ocr_2017_05_17/model.ckpt-399731.index')
print tf.train.get_checkpoint_state('../attention_ocr_2017_05_17/model.ckpt-399731')

returns:

Ture
Ture
None

A very simple test is attempted:

saver = tf.train.import_meta_graph('../attention_ocr_2017_05_17/model.ckpt-399731.meta')
with tf.Session() as sess:
    print os.path.exists('./attention_ocr_2017_05_17/model.ckpt-399731.meta')
    print tf.train.get_checkpoint_state('../attention_ocr_2017_05_17/model.ckpt-399731')
    saver.restore(sess,'../attention_ocr_2017_05_17/model.ckpt-399731')

returns no error, but still not restored:

2017-08-06 16:24:41.346086: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
True
2017-08-06 16:24:41.346124: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-06 16:24:41.346129: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-06 16:24:41.346133: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
None
INFO:tensorflow:Restoring parameters from ../attention_ocr_2017_05_17/model.ckpt-399731
INFO 2017-08-06 16:24:41.000354: tf_logging.py: 82 Restoring parameters from ../attention_ocr_2017_05_17/model.ckpt-399731

Process finished with exit code 0

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

import tensorflow as tf
import os
from tensorflow.python.platform import flags
import matplotlib.image as mpimg
import common_flags

FLAGS = flags.FLAGS
common_flags.define()

# yapf: disable
flags.DEFINE_integer('num_batches', 100,
                     'Number of batches to run eval for.')

flags.DEFINE_string('eval_log_dir', '/tmp/attention_ocr/eval',
                    'Directory where the evaluation results are saved to.')

flags.DEFINE_integer('eval_interval_secs', 60,
                     'Frequency in seconds to run evaluations.')

flags.DEFINE_integer('number_of_steps', None,
                     'Number of times to run evaluation.')

# fake a simple test image

raw_image_data = mpimg.imread('A4A8A5910A355-cvt.jpg').reshape((1,150,600,3))
images_placeholder = tf.placeholder(tf.float32,shape = (1,150, 600, 3),name='img_data')

if not tf.gfile.Exists(FLAGS.eval_log_dir):
    tf.gfile.MakeDirs(FLAGS.eval_log_dir)
dataset = common_flags.create_dataset(split_name=FLAGS.split_name)
model = common_flags.create_model(dataset.num_char_classes,
                                    dataset.max_sequence_length,
                                    dataset.num_of_views, dataset.null_code)
endpoints = model.create_base(images_placeholder, labels_one_hot=None)

# start loading attention_ocr model

saver = tf.train.import_meta_graph('../attention_ocr_2017_05_17/model.ckpt-399731.meta')

with tf.Session() as sess:
    # init without checkpoint variables and predict
    init = tf.global_variables_initializer()
    sess.run(init)
    predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
    print predictions

    # restore from checkpoint then predict
    print os.path.exists('./attention_ocr_2017_05_17/model.ckpt-399731.meta')
    print tf.train.get_checkpoint_state('../attention_ocr_2017_05_17/model.ckpt-399731')
    saver.restore(sess,'../attention_ocr_2017_05_17/model.ckpt-399731')
    predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
    print predictions

INFO 2017-08-06 16:48:44.000554: fsns.py: 130 Using FSNS dataset split_name=train dataset_dir=/Users/liuhuichuan/PycharmProjects/models/attention_ocr/python/datasets/data/fsns
DEBUG 2017-08-06 16:48:44.000556: model.py: 343 images: Tensor("img_data:0", shape=(1, 150, 600, 3), dtype=float32)
DEBUG 2017-08-06 16:48:44.000561: model.py: 348 Views=4 single view: Tensor("AttentionOcr_v1/split:0", shape=(1, 150, 150, 3), dtype=float32)
DEBUG 2017-08-06 16:48:44.000561: model.py: 191 Using final_endpoint=Mixed_5d
DEBUG 2017-08-06 16:48:46.000492: model.py: 191 Using final_endpoint=Mixed_5d
DEBUG 2017-08-06 16:48:47.000546: model.py: 191 Using final_endpoint=Mixed_5d
DEBUG 2017-08-06 16:48:48.000684: model.py: 191 Using final_endpoint=Mixed_5d
DEBUG 2017-08-06 16:48:49.000862: model.py: 354 Conv tower: Tensor("AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Mixed_5d/concat:0", shape=(1, 16, 16, 288), dtype=float32)
DEBUG 2017-08-06 16:48:49.000862: model.py: 357 Conv tower w/ encoded coordinates: Tensor("AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Mixed_5d/concat:0", shape=(1, 16, 16, 288), dtype=float32)
DEBUG 2017-08-06 16:48:49.000869: model.py: 360 Pooled views: Tensor("AttentionOcr_v1/pool_views_fn/STCK/Reshape:0", shape=(1, 1024, 288), dtype=float32)
DEBUG 2017-08-06 16:48:49.000869: sequence_layers.py: 421 Use AttentionWithAutoregression as a layer class
DEBUG 2017-08-06 16:48:53.000099: model.py: 363 chars_logit: Tensor("AttentionOcr_v1/sequence_logit_fn/SQLR/concat:0", shape=(1, 37, 134), dtype=float32)
2017-08-06 16:50:05.943512: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-06 16:50:05.943528: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-06 16:50:05.943532: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-06 16:50:05.943537: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
INFO:tensorflow:Restoring parameters from ../attention_ocr_2017_05_17/model.ckpt-399731
INFO 2017-08-06 16:50:29.000024: tf_logging.py: 82 Restoring parameters from ../attention_ocr_2017_05_17/model.ckpt-399731
[[  0   0   0 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123
  123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123
  123]]
True
None
[[  0   0   0 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123
  123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123 123
  123]]

Process finished with exit code 0

alexgorban commented 7 years ago

There are also multiple reports of the related problems: https://stackoverflow.com/questions/45551656 https://stackoverflow.com/questions/45151907

I'll take a look on it shortly.

alexgorban commented 7 years ago

Recent change in tensor names for the attention decoder cased all existing checkpoints to break. Please use the latest version of the checkpoint_convert.py script to fix them.

You also can take a look on the just created demo_inference.py (see the pull request above).

tatatodd commented 7 years ago

Thanks for the analysis @alexgorban !

@HuichuanLiu Please report back describing whether this works for you. Thanks!

alexgorban commented 7 years ago

Here is the updated checkpoint http://download.tensorflow.org/models/attention_ocr_2017_08_09.tar.gz Please let me know if it fixes the problem.

HuichuanLiu commented 7 years ago

Sure, will respond in 2 days.

HuichuanLiu commented 7 years ago

The new checkpoint doesn't work:

saver = tf.train.import_meta_graph('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
with tf.Session() as sess:
    # init without checkpoint variables and predict
    init = tf.global_variables_initializer()

    if True:
        sess.run(init)
        predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
        print predictions

    # restore from checkpoint then predict
    print '.meta file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print '.index file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print '.data file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print 'Checkpoint available?',tf.train.get_checkpoint_state('./attention_ocr_2017_08_09/model.ckpt-399731')
    saver.restore(sess,'./attention_ocr_2017_08_09/model.ckpt-399731')
    predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
    print predictions

Returns:

[[42 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
  30 30 30 30 30 30 30 30 30 30 30 30 30]]
.meta file exists?  True
.index file exists?  True
.data file exists?  True
Checkpoint available? None
INFO:tensorflow:Restoring parameters from ./attention_ocr_2017_08_09/model.ckpt-399731
INFO 2017-08-10 14:08:06.000200: tf_logging.py: 82 Restoring parameters from ./attention_ocr_2017_08_09/model.ckpt-399731
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-16-3d42ad864f96> in <module>()
     13     print '.data file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
     14     print 'Checkpoint available?',tf.train.get_checkpoint_state('./attention_ocr_2017_08_09/model.ckpt-399731')
---> 15     saver.restore(sess,'./attention_ocr_2017_08_09/model.ckpt-399731')
     16     predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
     17     print predictions

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.pyc in restore(self, sess, save_path)
   1546     logging.info("Restoring parameters from %s", save_path)
   1547     sess.run(self.saver_def.restore_op_name,
-> 1548              {self.saver_def.filename_tensor_name: save_path})
   1549 
   1550   @staticmethod

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    787     try:
    788       result = self._run(None, fetches, feed_dict, options_ptr,
--> 789                          run_metadata_ptr)
    790       if run_metadata:
    791         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
    995     if final_fetches or final_targets:
    996       results = self._do_run(handle, final_targets, final_fetches,
--> 997                              feed_dict_string, options, run_metadata)
    998     else:
    999       results = []

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1130     if handle is None:
   1131       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1132                            target_list, options, run_metadata)
   1133     else:
   1134       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1150         except KeyError:
   1151           pass
-> 1152       raise type(e)(node_def, op, message)
   1153 
   1154   def _extend_graph(self):

NotFoundError: Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/weights/Momentum not found in checkpoint
     [[Node: save/RestoreV2_175 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_175/tensor_names, save/RestoreV2_175/shape_and_slices)]]

Caused by op u'save/RestoreV2_175', defined at:
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-8a1327ee2e19>", line 1, in <module>
    saver = tf.train.import_meta_graph('../attention_ocr_2017_05_17/model.ckpt-399731.meta')
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1686, in import_meta_graph
    **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 504, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 311, in import_graph_def
    op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/weights/Momentum not found in checkpoint
     [[Node: save/RestoreV2_175 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_175/tensor_names, save/RestoreV2_175/shape_and_slices)]]

Seems like some problem with decoding?

HuichuanLiu commented 7 years ago

The same with convert_checkpoint.py

saver = tf.train.import_meta_graph('./my_converted_checkpoint/model/model.ckpt-399731.meta')
with tf.Session() as sess:
    # init without checkpoint variables and predict
    init = tf.global_variables_initializer()

    if True:
        sess.run(init)
        predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
        print predictions

    # restore from checkpoint then predict
    print '.meta file exists? ',os.path.exists('./my_converted_checkpoint/model/model.ckpt-399731.meta')
    print '.index file exists? ',os.path.exists('./my_converted_checkpoint/model/model.ckpt-399731.meta')
    print '.data file exists? ',os.path.exists('./my_converted_checkpoint/model/model.ckpt-399731.meta')
    print 'Checkpoint available?',tf.train.get_checkpoint_state('./my_converted_checkpoint/model/model.ckpt-399731')
    saver.restore(sess,'./my_converted_checkpoint/model/model.ckpt-399731')
    predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
    print predictions

Returns:

[[36 36 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14
  14 14 14 14 14 14 14 14 14 14 14 14 14]]
.meta file exists?  True
.index file exists?  True
.data file exists?  True
Checkpoint available? None
INFO:tensorflow:Restoring parameters from ./my_converted_checkpoint/model/model.ckpt-399731
INFO 2017-08-10 14:17:20.000871: tf_logging.py: 82 Restoring parameters from ./my_converted_checkpoint/model/model.ckpt-399731
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-18-7eefd1a500b8> in <module>()
     13     print '.data file exists? ',os.path.exists('./my_converted_checkpoint/model/model.ckpt-399731.meta')
     14     print 'Checkpoint available?',tf.train.get_checkpoint_state('./my_converted_checkpoint/model/model.ckpt-399731')
---> 15     saver.restore(sess,'./my_converted_checkpoint/model/model.ckpt-399731')
     16     predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
     17     print predictions

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.pyc in restore(self, sess, save_path)
   1546     logging.info("Restoring parameters from %s", save_path)
   1547     sess.run(self.saver_def.restore_op_name,
-> 1548              {self.saver_def.filename_tensor_name: save_path})
   1549 
   1550   @staticmethod

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    787     try:
    788       result = self._run(None, fetches, feed_dict, options_ptr,
--> 789                          run_metadata_ptr)
    790       if run_metadata:
    791         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
    995     if final_fetches or final_targets:
    996       results = self._do_run(handle, final_targets, final_fetches,
--> 997                              feed_dict_string, options, run_metadata)
    998     else:
    999       results = []

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1130     if handle is None:
   1131       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1132                            target_list, options, run_metadata)
   1133     else:
   1134       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1150         except KeyError:
   1151           pass
-> 1152       raise type(e)(node_def, op, message)
   1153 
   1154   def _extend_graph(self):

NotFoundError: Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/weights/Momentum not found in checkpoint
     [[Node: save/RestoreV2_175 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_175/tensor_names, save/RestoreV2_175/shape_and_slices)]]

Caused by op u'save/RestoreV2_175', defined at:
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-8a1327ee2e19>", line 1, in <module>
    saver = tf.train.import_meta_graph('../attention_ocr_2017_05_17/model.ckpt-399731.meta')
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1686, in import_meta_graph
    **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 504, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 311, in import_graph_def
    op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/weights/Momentum not found in checkpoint
     [[Node: save/RestoreV2_175 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_175/tensor_names, save/RestoreV2_175/shape_and_slices)]]

HuichuanLiu commented 7 years ago

Try to get rid of the init test:

saver = tf.train.import_meta_graph('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
with tf.Session() as sess:
    # init without checkpoint variables and predict

    if False:
        init = tf.global_variables_initializer()
        sess.run(init)
        predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
        print predictions

    # restore from checkpoint then predict
    print '.meta file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print '.index file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print '.data file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
    print 'Checkpoint available?',tf.train.get_checkpoint_state('./attention_ocr_2017_08_09/model.ckpt-399731')
    saver.restore(sess,'./attention_ocr_2017_08_09/model.ckpt-399731')
    predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
    print predictions

Returns:

.meta file exists?  True
.index file exists?  True
.data file exists?  True
Checkpoint available? None
INFO:tensorflow:Restoring parameters from ./attention_ocr_2017_08_09/model.ckpt-399731
INFO 2017-08-10 14:24:52.000959: tf_logging.py: 82 Restoring parameters from ./attention_ocr_2017_08_09/model.ckpt-399731
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-20-ed9432233500> in <module>()
     14     print '.data file exists? ',os.path.exists('./attention_ocr_2017_08_09/model.ckpt-399731.meta')
     15     print 'Checkpoint available?',tf.train.get_checkpoint_state('./attention_ocr_2017_08_09/model.ckpt-399731')
---> 16     saver.restore(sess,'./attention_ocr_2017_08_09/model.ckpt-399731')
     17     predictions = sess.run(endpoints.predicted_chars, feed_dict={images_placeholder: raw_image_data})
     18     print predictions

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.pyc in restore(self, sess, save_path)
   1546     logging.info("Restoring parameters from %s", save_path)
   1547     sess.run(self.saver_def.restore_op_name,
-> 1548              {self.saver_def.filename_tensor_name: save_path})
   1549 
   1550   @staticmethod

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    787     try:
    788       result = self._run(None, fetches, feed_dict, options_ptr,
--> 789                          run_metadata_ptr)
    790       if run_metadata:
    791         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
    995     if final_fetches or final_targets:
    996       results = self._do_run(handle, final_targets, final_fetches,
--> 997                              feed_dict_string, options, run_metadata)
    998     else:
    999       results = []

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1130     if handle is None:
   1131       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1132                            target_list, options, run_metadata)
   1133     else:
   1134       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1150         except KeyError:
   1151           pass
-> 1152       raise type(e)(node_def, op, message)
   1153 
   1154   def _extend_graph(self):

NotFoundError: Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/Attention_0/biases not found in checkpoint
     [[Node: save/RestoreV2_156 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_156/tensor_names, save/RestoreV2_156/shape_and_slices)]]

Caused by op u'save/RestoreV2_156', defined at:
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-8a1327ee2e19>", line 1, in <module>
    saver = tf.train.import_meta_graph('../attention_ocr_2017_05_17/model.ckpt-399731.meta')
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1686, in import_meta_graph
    **kwargs)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 504, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 311, in import_graph_def
    op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/liuhuichuan/anaconda2/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key AttentionOcr_v1/sequence_logit_fn/SQLR/LSTM/attention_decoder/Attention_0/biases not found in checkpoint
     [[Node: save/RestoreV2_156 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_156/tensor_names, save/RestoreV2_156/shape_and_slices)]]

Obviously the sess.run(init) affects the graphs/weights. Removing this part leads to another error.

BrianGong commented 7 years ago

I checked out the code from https://github.com/alexgorban/models/tree/dff0f0c1143876c374f2f6887fb7a59281344a8f,and run demo_inference.py(https://github.com/alexgorban/models/blob/dff0f0c1143876c374f2f6887fb7a59281344a8f/attention_ocr/python/demo_inference.py).

get the error like this:

INFO 2017-08-15 17:50:09.000364: fsns.py: 130 Using FSNS dataset split_name=train dataset_dir=/Users/cloudin/Downloads/models-dff0f0c1143876c374f2f6887fb7a59281344a8f/attention_ocr/python/datasets/data/fsns
WARNING 2017-08-15 17:50:09.000366: fsns.py: 79 incorrect charset file. line #0: 0

WARNING 2017-08-15 17:50:09.000367: fsns.py: 79 incorrect charset file. line #139: 

DEBUG 2017-08-15 17:50:09.000369: model.py: 352 images: Tensor("Placeholder:0", shape=(32, 150, 600, 3), dtype=float32)
DEBUG 2017-08-15 17:50:09.000374: model.py: 357 Views=4 single view: Tensor("AttentionOcr_v1/split:0", shape=(32, 150, 150, 3), dtype=float32)
DEBUG 2017-08-15 17:50:09.000374: model.py: 200 Using final_endpoint=Mixed_5d
DEBUG 2017-08-15 17:50:11.000241: model.py: 200 Using final_endpoint=Mixed_5d
DEBUG 2017-08-15 17:50:12.000366: model.py: 200 Using final_endpoint=Mixed_5d
DEBUG 2017-08-15 17:50:13.000353: model.py: 200 Using final_endpoint=Mixed_5d
DEBUG 2017-08-15 17:50:14.000504: model.py: 363 Conv tower: Tensor("AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Mixed_5d/concat:0", shape=(32, 16, 16, 288), dtype=float32)
DEBUG 2017-08-15 17:50:14.000504: model.py: 366 Conv tower w/ encoded coordinates: Tensor("AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Mixed_5d/concat:0", shape=(32, 16, 16, 288), dtype=float32)
DEBUG 2017-08-15 17:50:14.000511: model.py: 369 Pooled views: Tensor("AttentionOcr_v1/pool_views_fn/STCK/Reshape:0", shape=(32, 1024, 288), dtype=float32)
DEBUG 2017-08-15 17:50:14.000512: sequence_layers.py: 421 Use AttentionWithAutoregression as a layer class
DEBUG 2017-08-15 17:50:17.000521: model.py: 372 chars_logit: Tensor("AttentionOcr_v1/sequence_logit_fn/SQLR/concat:0", shape=(32, 37, 133), dtype=float32)
Reading /Users/cloudin/OCR/attention_ocr_2017_08_09/A4A8A5910A355-cvt_0.jpg
2017-08-15 17:50:17.643895: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-15 17:50:17.643921: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-15 17:50:17.643927: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-15 17:50:17.643931: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
INFO 2017-08-15 17:50:20.000606: model.py: 575 Restoring checkpoint(s)
Traceback (most recent call last):
  File "demo_inference.py", line 86, in <module>
    tf.app.run()
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "demo_inference.py", line 82, in main
    feed_dict={images_placeholder: images_data})
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights
     [[Node: AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights/read = Identity[T=DT_FLOAT, _class=["loc:@AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights)]]

Caused by op u'AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights/read', defined at:
  File "demo_inference.py", line 86, in <module>
    tf.app.run()
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "demo_inference.py", line 74, in main
    FLAGS.dataset_name)
  File "demo_inference.py", line 66, in load_model
    endpoints = model.create_base(images_placeholder, labels_one_hot=None)
  File "/Users/cloudin/Downloads/models-dff0f0c1143876c374f2f6887fb7a59281344a8f/attention_ocr/python/model.py", line 361, in create_base
    for i, v in enumerate(views)
  File "/Users/cloudin/Downloads/models-dff0f0c1143876c374f2f6887fb7a59281344a8f/attention_ocr/python/model.py", line 206, in conv_tower_fn
    images, final_endpoint=mparams.final_endpoint)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/inception_v3.py", line 111, in inception_v3_base
    net = layers.conv2d(inputs, depth(32), [3, 3], stride=2, scope=end_point)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 947, in convolution
    outputs = layer.apply(inputs)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 492, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 434, in __call__
    self.build(input_shapes[0])
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/layers/convolutional.py", line 137, in build
    dtype=self.dtype)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 374, in add_variable
    trainable=trainable and self.trainable)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable
    use_resource=use_resource, custom_getter=custom_getter)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable
    use_resource=use_resource, custom_getter=custom_getter)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 360, in get_variable
    validate_shape=validate_shape, use_resource=use_resource)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1370, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1362, in _model_variable_getter
    custom_getter=getter, use_resource=use_resource)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 261, in model_variable
    use_resource=use_resource)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 216, in variable
    use_resource=use_resource)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter
    use_resource=use_resource)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 725, in _get_single_variable
    validate_shape=validate_shape)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 200, in __init__
    expected_shape=expected_shape)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 319, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1303, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/cloudin/anaconda/envs/OCR/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights
     [[Node: AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights/read = Identity[T=DT_FLOAT, _class=["loc:@AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights"], _device="/job:localhost/replica:0/task:0/cpu:0"](AttentionOcr_v1/conv_tower_fn/INCE/InceptionV3/Conv2d_1a_3x3/weights)]]

and this is my code ,and use the check point(http://download.tensorflow.org/models/attention_ocr_2017_08_09.tar.gz) :

FLAGS = flags.FLAGS
common_flags.define()

flags.DEFINE_string('image_path_pattern', '/Users/brian/OCR/attention_ocr_2017_08_09/A4A8A5910A355-cvt_%d.jpg',
                    'A file pattern with a placeholder for the image index.')

def get_dataset_image_size(dataset_name):
  ds_module = getattr(datasets, dataset_name)
  height, width, _ = ds_module.DEFAULT_CONFIG['image_shape']
  return width, height

def load_images(file_pattern, batch_size, dataset_name):
  width, height = get_dataset_image_size(dataset_name)
  images_actual_data = np.ndarray(shape=(batch_size, height, width, 3),
                                  dtype='float32')
  for i in range(1):
    path = file_pattern % i
    print("Reading %s" % path)
    pil_image = PIL.Image.open(tf.gfile.GFile(path))
    images_actual_data[i, ...] = np.asarray(pil_image)
  return images_actual_data

def load_model(checkpoint, batch_size, dataset_name):
  width, height = get_dataset_image_size(dataset_name)
  dataset = common_flags.create_dataset(split_name=FLAGS.split_name)
  model = common_flags.create_model(
      num_char_classes=dataset.num_char_classes,
      seq_length=dataset.max_sequence_length,
      num_views=dataset.num_of_views,
      null_code=dataset.null_code,
      charset=dataset.charset)
  images_placeholder = tf.placeholder(tf.float32,
                                      shape=[batch_size, height, width, 3])
  endpoints = model.create_base(images_placeholder, labels_one_hot=None)
  init_fn = model.create_init_fn_to_restore(checkpoint)
  return images_placeholder, endpoints, init_fn

def main(_):
  images_placeholder, endpoints, init_fn = load_model(FLAGS.checkpoint,
                                                      FLAGS.batch_size,
                                                      FLAGS.dataset_name)
  images_data = load_images(FLAGS.image_path_pattern, FLAGS.batch_size,
                            FLAGS.dataset_name)
  with tf.Session() as sess:
    tf.tables_initializer().run()  # required by the CharsetMapper
    init_fn(sess)

    predictions = sess.run(endpoints.predicted_text,
                           feed_dict={images_placeholder: images_data})
  print("Predicted strings:")

if __name__ == '__main__':
  tf.app.run()

aselle commented 7 years ago

@alexgorban, PTAL.

alexgorban commented 7 years ago

I'm sorry for the terribly delayed response.

To @HuichuanLiu:

The new checkpoint doesn't work: saver = tf.train.import_meta_graph('./attention_ocr_2017_08_09/model.ckpt-399731.meta')

this will load only metagraph, but you need to load weights as well. There is a handy function for that: init_fn = model.create_init_fn_to_restore(checkpoint) init_fn(sess) Refer to demo_inference.py for more details.

To @BrianGong:

I checked out the code from alexgorban/models/ and run demo_inference.py get the error like this: Perhaps you didn't specified the --checkpoint flag, it was missing (#2565) in the usage example. Please try:

python demo_inference.py \ --batch_size=32 \ --image_path_pattern=./datasets/data/fsns/temp/fsnstrain%02d.png \ --checkpoint=model.ckpt-399731

(it should work, I've just double checked it with TensorFlow 1.3.0)

To @aselle: I think we can close this issue.

tensorflow / models