Model keeps accumulating the new and the restored variables after restoring from a checkpoint resulting in bigger model size.

keithito / tacotron

A TensorFlow implementation of Google's Tacotron speech synthesis with pre-trained model (unofficial)

MIT License

2.95k stars 959 forks source link

Model keeps accumulating the new and the restored variables after restoring from a checkpoint resulting in bigger model size. #268

Open shreshtha-pankaj opened 5 years ago

shreshtha-pankaj commented 5 years ago

When you restore the model from a checkpoint it retains the new variables which are created as part of the initialization of the model during the restore in the tf.session along with the restored variables.
So when the restored model is saved again the size of the checkpoint doubles or increases by the previous checkpoint size.
If we do this several times the size of the checkpoint becomes too large to fit in the memory.

shreshtha-pankaj commented 5 years ago

Image showing duplicate variables in the session