search

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
https://nvidia.github.io/OpenSeq2Seq
Apache License 2.0
1.54k stars 372 forks source link

Saving with freeze_variables_regex do not save frozen layers #466

Open phypan11 opened 5 years ago

phypan11 commented 5 years ago

I am trying to use freeze_variables_regex to freeze some layers. But when it saves the checkpoint, only the unfrozen ones are saved. Thus when I try to train all layers after training part of the layers, it does not work. How can I fix this problem?

What I found so far is that when I freeze the graph and make a checkpoint and then try to load into full model, It gets an error not able to find 'adam' layer. For instance:

tensorflow.python.framework.errors_impl.NotFoundError: Key Loss_Optimization/ForwardPass/tacotron2_encoder/EncoderEmbeddingMatrix/Adam not found in checkpoint

blisc commented 5 years ago

Are you also specifying load_model?