File "/home/pajain34/parser/nmt/nmt/nmt.py", line 696, in main run_main(FLAGS, default_hparams, train_fn, inference_fn) File "/home/pajain34/parser/nmt/nmt/nmt.py", line 689, in run_main train_fn(hparams, target_session=target_session) File "/home/pajain34/parser/nmt/nmt/train.py", line 542, in train feed_dict={train_model.skip_count_placeholder: 0}) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run run_metadata_ptr) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1140, in _run feed_dict_tensor, options, run_metadata) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run run_metadata) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Run call was cancelled
I understand this may not be a bug, and I have posted same question in stackoverflow. Any help or pointers will be extremely helpful.
I am trying to use partial run with nmt-1.4 setup. Following is the change I have made, here: https://github.com/tensorflow/nmt/blob/tf-1.4/nmt/model.py#L227
partial_setup = sess.partial_run_setup([self.train_super_op_dummy_reward, self.train_loss, self.predict_count, self.train_summary, self.global_step, self.word_count, self.batch_size]) step_loss = sess.partial_run(partial_setup, self.train_loss) r = sess.partial_run(partial_setup, self.train_super_op_dummy_reward) step_predict_count, step_summary, global_step, step_word_count, batch_size = sess.partial_run(partial_setup, [self.predict_count,self.train_summary, self.global_step, self.word_count,self.batch_size]) return r, step_loss, step_predict_count, step_summary, global_step, step_word_count, batch_size
I get this error after one epoch
File "/home/pajain34/parser/nmt/nmt/nmt.py", line 696, in main run_main(FLAGS, default_hparams, train_fn, inference_fn) File "/home/pajain34/parser/nmt/nmt/nmt.py", line 689, in run_main train_fn(hparams, target_session=target_session) File "/home/pajain34/parser/nmt/nmt/train.py", line 542, in train feed_dict={train_model.skip_count_placeholder: 0}) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run run_metadata_ptr) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1140, in _run feed_dict_tensor, options, run_metadata) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run run_metadata) File "/home/pajain34/virtual-env/tf1.7/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Run call was cancelled
I understand this may not be a bug, and I have posted same question in stackoverflow. Any help or pointers will be extremely helpful.