search

suriyadeepan / practical_seq2seq

A simple, minimal wrapper for tensorflow's seq2seq module, for experimenting with datasets rapidly
http://suriyadeepan.github.io/2016-12-31-practical-seq2seq/
GNU General Public License v3.0
569 stars 270 forks source link

Error running the twitter dataset bot after reloading checkpoint. #22

Open vaibhavravi opened 7 years ago

vaibhavravi commented 7 years ago

I am getting the error shown below after I run the model.predict() line in the twitter chatbot ipynb.

After some searching online I found that this happens when the variables are not initialized, but I am just trying to run your pre-trained model to see how it works.

Is there something I should be doing that's different ? Should I train before I run?

---------------------------------------------------------------------------
FailedPreconditionError                   Traceback (most recent call last)
/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
    971     try:
--> 972       return fn(*args)
    973     except errors.OpError as e:

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
    953                                  feed_dict, fetch_list, target_list,
--> 954                                  status, run_metadata)
    955 

/Users/ybow_93/anaconda/lib/python3.5/contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/errors.py in raise_exception_on_not_ok_status()
    462           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 463           pywrap_tensorflow.TF_GetCode(status))
    464   finally:

FailedPreconditionError: Attempting to use uninitialized value decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias
     [[Node: decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias/read = Identity[T=DT_FLOAT, _class=["loc:@decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias"], _device="/job:localhost/replica:0/task:0/cpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias)]]

During handling of the above exception, another exception occurred:

FailedPreconditionError                   Traceback (most recent call last)
<ipython-input-32-6ce101760870> in <module>()
      1 #sess.run(tf.initialize_all_variables())
      2 input_ = test_batch_gen.__next__()[0]
----> 3 output = model.predict(sess, input_)
      4 print(output.shape)

/Users/ybow_93/Deep and Machine Learning/Retrieval Based Chatbot/practical_seq2seq-master/seq2seq_wrapper.py in predict(self, sess, X)
    173         feed_dict = {self.enc_ip[t]: X[t] for t in range(self.xseq_len)}
    174         feed_dict[self.keep_prob] = 1.
--> 175         dec_op_v = sess.run(self.decode_outputs_test, feed_dict)
    176         # dec_op_v is a list; also need to transpose 0,1 indices
    177         #  (interchange batch_size and timesteps dimensions

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    715     try:
    716       result = self._run(None, fetches, feed_dict, options_ptr,
--> 717                          run_metadata_ptr)
    718       if run_metadata:
    719         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    913     if final_fetches or final_targets:
    914       results = self._do_run(handle, final_targets, final_fetches,
--> 915                              feed_dict_string, options, run_metadata)
    916     else:
    917       results = []

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
    963     if handle is None:
    964       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
--> 965                            target_list, options, run_metadata)
    966     else:
    967       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
    983         except KeyError:
    984           pass
--> 985       raise type(e)(node_def, op, message)
    986 
    987   def _extend_graph(self):

FailedPreconditionError: Attempting to use uninitialized value decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias
     [[Node: decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias/read = Identity[T=DT_FLOAT, _class=["loc:@decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias"], _device="/job:localhost/replica:0/task:0/cpu:0"](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias)]]

Caused by op 'decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias/read', defined at:
  File "/Users/ybow_93/anaconda/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/ybow_93/anaconda/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 474, in start
    ioloop.IOLoop.instance().start()
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tornado/ioloop.py", line 887, in start
    handler_func(fd_obj, events)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 276, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell
    handler(stream, idents, msg)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 390, in execute_request
    user_expressions, allow_stdin)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 501, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-29-27a584e39e51>", line 7, in <module>
    num_layers=3
  File "/Users/ybow_93/Deep and Machine Learning/Retrieval Based Chatbot/practical_seq2seq-master/seq2seq_wrapper.py", line 79, in __init__
    __graph__()
  File "/Users/ybow_93/Deep and Machine Learning/Retrieval Based Chatbot/practical_seq2seq-master/seq2seq_wrapper.py", line 58, in __graph__
    xvocab_size, yvocab_size, emb_dim)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/seq2seq.py", line 357, in embedding_rnn_seq2seq
    feed_previous=feed_previous)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/seq2seq.py", line 280, in embedding_rnn_decoder
    loop_function=loop_function)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/seq2seq.py", line 146, in rnn_decoder
    output, state = cell(inp, state)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/rnn_cell.py", line 587, in __call__
    projected = _linear(output, self._output_size, True)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/rnn_cell.py", line 914, in _linear
    bias_start, dtype=dtype))
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 1022, in get_variable
    custom_getter=custom_getter)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 849, in get_variable
    custom_getter=custom_getter)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 345, in get_variable
    validate_shape=validate_shape)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 330, in _true_getter
    caching_device=caching_device, validate_shape=validate_shape)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py", line 676, in _get_single_variable
    validate_shape=validate_shape)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variables.py", line 215, in __init__
    dtype=dtype)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/variables.py", line 327, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1128, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/ybow_93/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias
     [ [ Node: decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias/read = Identity[T=DT_FLOAT, _class=[ "loc:@decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias"], _device="/job:localhost/replica:0/task:0/cpu:0" ](decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/OutputProjectionWrapper/Linear/Bias ) ] ]
mandarbandodekar commented 6 years ago

I am having the exact same error!! Did u resolve it?

GreatKarollo commented 6 years ago

have you guys manage to find the solution ? I tried initialize_all_variables and graph root but no success so far ? Do you think it is to do with TF 1.4 breaking compatibly with previous vs?

nunezpaul commented 6 years ago

Hey I was having this problem too. The problem is you're likely not importing the ckpt files.

Turns out that running the code straight will not give you an error if you haven't actually loaded the checkpoint file since the code will skip over the loading if it doesn't find the files.

 def restore_last_session(self):
    saver = tf.train.Saver()
    # create a session
    sess = tf.Session()
    # get checkpoint state
    ckpt = tf.train.get_checkpoint_state(self.ckpt_path) 
    # restore session
    if ckpt and ckpt.model_checkpoint_path: [<<<<<<<<<<<<<]
        saver.restore(sess, ckpt.model_checkpoint_path)
    # return to user
    return sess

To fix it this issue, you need to do either one of the following after pulling and decompressing the model.

1) make sure to either have all the ckpt files directly in the ckpt folder 2) modify the ckpt_path (line 29 in chatbot.py) to be

`ckpt = 'ckpt/seq2seq_twitter_1024x3h_i43000'`

*assuming that your uncompressed folder is named the same as mine. If not then change seq2seq_twitter_1024x3h_i43000 to whatever you've named it.

That solved my problem and will likely fix yours.

singhgurjyot commented 6 years ago

Hey nunezpaul, I did what you said but it still gives me some NotFoundError, I am running python 3.5.4 and tensorflow 1.0.0

Here is the error: NotFoundError (see above for traceback): Key decoder/embedding_rnn_seq2seq/embedding_rnn_decoder/rnn_decoder/multi_rnn_cell/cell_1/basic_lstm_cell/biases/Adam_1 not found in checkpoint [[Node: save/RestoreV2_13 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_13/tensor_names, save/RestoreV2_13/shape_and_slices)]]

LukeThomas-AA commented 4 years ago

^^^^