search

uds-lsv / TF-NNLM-TK

A toolkit for neural language modeling using Tensorflow including basic models like RNNs and LSTMs as well as more advanced models.
Other
20 stars 2 forks source link

Unable to restore the trained model #2

Closed tastyminerals closed 6 years ago

tastyminerals commented 6 years ago

I have successfully trained a sequential classifier model using TF-NNLM-TK code as a baseline. The model was saved using the following code:

# save the final model
model_path = os.path.join(config.save, "model.ckpt")
saver.save(session, "saved_models")

The model is then stored in saved_models/ dir as a set of files:

checkpoint  
events.out.tfevents.1521025726.pavel-pc  
model.ckpt.data-00000-of-00001  
model.ckpt.index  
model.ckpt.meta

Attempting to restore it 

# starting default session
with tf.Session() as session:
    saver = tf.train.Saver(tf.global_variables())
    saver.restore(session, "saved_model/model.ckpt.data-00000-of-00001")

throws DataLossError

DataLossError (see above for traceback): Unable to open table file saved_model/model.ckpt.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
     [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]
     [[Node: save/RestoreV2_6/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_20_save/RestoreV2_6", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

tensorflow-gpu: 1.5.0 running on Linux

tastyminerals commented 6 years ago

Resolved it by using a different file name for restoring:

# starting default session
with tf.Session() as session:
    saver = tf.train.Saver(tf.global_variables())
    saver.restore(session, "saved_model/model.ckpt")