2018-08-13 08:34:00.525937: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key adversary/fully_connected/biases not found in checkpoint
Traceback (most recent call last):
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key adversary/fully_connected/biases not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 125, in <module>
main(args)
File "main.py", line 117, in main
number_trajs=10, stochastic_policy=args.stochastic_policy)
File "/home/huang/gail-tf/gailtf/algo/trpo_mpi.py", line 400, in evaluate
U.load_state(load_model_path)
File "/home/huang/gail-tf/gailtf/baselines/common/tf_util.py", line 280, in load_state
saver.restore(get_session(), fname)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1775, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1140, in _run
feed_dict_tensor, options, run_metadata)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
run_metadata)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key adversary/fully_connected/biases not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
Caused by op 'save/RestoreV2', defined at:
File "main.py", line 125, in <module>
main(args)
File "main.py", line 117, in main
number_trajs=10, stochastic_policy=args.stochastic_policy)
File "/home/huang/gail-tf/gailtf/algo/trpo_mpi.py", line 400, in evaluate
U.load_state(load_model_path)
File "/home/huang/gail-tf/gailtf/baselines/common/tf_util.py", line 279, in load_state
else: saver = tf.train.Saver()
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1311, in __init__
self.build()
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1320, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1357, in _build
build_save=build_save, build_restore=build_restore)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 809, in _build_internal
restore_sequentially, reshape)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 448, in _AddRestoreOps
restore_sequentially)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 860, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1458, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
op_def=op_def)
File "/home/huang/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
NotFoundError (see above for traceback): Key adversary/fully_connected/biases not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True
$
python main.py --task evaluate --stochastic_policy True --expert_path gailtf/baselines/ppo1/stochastic.ppo.Hopper.0.00.pkl --load_model_path gailtf/baselines/ppo1/checkpoint/ppo.Hopper.0.00/ppo.Hopper.0.00-400
What is Key adversary means ?