search

shehzaadzd / MINERVA

Meandering In Networks of Entities to Reach Verisimilar Answers
Apache License 2.0
310 stars 87 forks source link

Unable to load the saved_model #21

Open kunal017 opened 5 years ago

kunal017 commented 5 years ago

I am running the model for countries_s1 model but getting the following error:

Executing python code/model/trainer.py --base_output_dir output/countries_s1/ --path_length 2 --hidden_size 25 --embedding_size 25     --batch_size 256 --beta 0.05 --Lambda 0.05 --use_entity_embeddings 1     --train_entity_embeddings 1 --train_relation_embeddings 1     --data_input_dir datasets/data_preprocessed/countries_s1/ --vocab_dir datasets/data_preprocessed/countries_s1/vocab --model_load_dir saved_models/countries_s1/model.ckpt --load_model 1 --total_iterations 1000 --nell_evaluation 0
HEY
Arguments:
                         LSTM_layers : 1
                              Lambda : 0.05
                     base_output_dir : output/countries_s1/
                          batch_size : 256
                                beta : 0.05
                        create_vocab : 0
                      data_input_dir : datasets/data_preprocessed/countries_s1/
                      embedding_size : 25
                          eval_every : 100
                               gamma : 1
                      grad_clip_norm : 5
                         hidden_size : 25
                          input_file : train.txt
                         input_files : ['datasets/data_preprocessed/countries_s1//train.txt']
                        l2_reg_const : 0.01
                       learning_rate : 0.001
                          load_model : True
                             log_dir : ./logs/
                       log_file_name : output/countries_s1//3f71_2_0.05_100_0.05/log.txt
                     max_num_actions : 200
                           model_dir : output/countries_s1//3f71_2_0.05_100_0.05/model/
                      model_load_dir : saved_models/countries_s1/model.ckpt
                     negative_reward : 0
                     nell_evaluation : 0
                        num_rollouts : 20
                          output_dir : output/countries_s1//3f71_2_0.05_100_0.05
                         output_file :
                         path_length : 2
                    path_logger_file : output/countries_s1//3f71_2_0.05_100_0.05
                                pool : max
                     positive_reward : 1.0
        pretrained_embeddings_action :
        pretrained_embeddings_entity :
                       test_rollouts : 100
                    total_iterations : 1000
             train_entity_embeddings : True
           train_relation_embeddings : True
               use_entity_embeddings : True
                           vocab_dir : datasets/data_preprocessed/countries_s1/vocab
INFO:root:reading vocab files...
01/28/2019 01:56:20 AM: [ reading vocab files... ]
INFO:root:Reading mid to name map
01/28/2019 01:56:20 AM: [ Reading mid to name map ]
INFO:root:Done..
01/28/2019 01:56:20 AM: [ Done.. ]
INFO:root:Total number of entities 273
01/28/2019 01:56:20 AM: [ Total number of entities 273 ]
INFO:root:Total number of relations 6
01/28/2019 01:56:20 AM: [ Total number of relations 6 ]
INFO:root:Skipping training
01/28/2019 01:56:20 AM: [ Skipping training ]
INFO:root:Loading model from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:20 AM: [ Loading model from saved_models/countries_s1/model.ckpt ]
Reading vocab...
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
2019-01-28 01:56:21.112771: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112794: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112801: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112806: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.988648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:06:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2019-01-28 01:56:21.988740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2019-01-28 01:56:21.988757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y
2019-01-28 01:56:21.988799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0)
INFO:root:Creating TF graph...
01/28/2019 01:56:22 AM: [ Creating TF graph... ]
INFO:root:TF Graph creation done..
01/28/2019 01:56:23 AM: [ TF Graph creation done.. ]
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:23 AM: [ Restoring parameters from saved_models/countries_s1/model.ckpt ]
2019-01-28 01:56:23.830867: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
2019-01-28 01:56:23.846931: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847249: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847266: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847290: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847342: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847358: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847437: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847753: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847603: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847794: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847881: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848095: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848317: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848375: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855353: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855395: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam not found in checkpoint
2019-01-28 01:56:23.855675: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855750: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855820: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856273: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856468: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856486: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856592: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856623: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856677: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
Traceback (most recent call last):
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 143, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_9', defined at:
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 137, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
shehzaadzd commented 5 years ago

It looks like the model was training with the train_entity_embeddingsparameter set as False. Could you change that in the config and rerun?

Albert-xj commented 2 years ago

@kunal017 Have you solved this problem? I also encountered NotFoundError (see above for traceback): Key policy_steps_unroll/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]] @shehzaadzd Can you help me?

shehzaadzd commented 2 years ago

Can you try training from scratch with the same parameters?

Albert-xj commented 2 years ago

Yes, retrain and saved model can be successfully loaded