search

Rayhane-mamah / Tacotron-2

DeepMind's Tacotron-2 Tensorflow implementation
MIT License
2.28k stars 905 forks source link

Checkpoint Key NotFoundError when doing WaveNet synthesizing #421

Closed JasonWei512 closed 5 years ago

JasonWei512 commented 5 years ago

When I run:

python synthesize.py --model='WaveNet'

I get a NotFoundError:

Key WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage not found in checkpoint

Full Error Log.txt

I modified some code of the lastest commit for Chinese language support, and trained Tacotron for 100K steps and WaveNet for 132K steps. My fork: https://github.com/JasonWei512/Tacotron-2-Chinese Compare: https://github.com/Rayhane-mamah/Tacotron-2/compare/master...JasonWei512:mandarin-biaobei

I ran print_tensors_in_checkpoint_file, and here's the tensor_name list in checkpoint file:

tensor_name in checkpoint > WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/final_convolution_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/final_convolution_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/final_convolution_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/final_convolution_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/input_convolution/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/input_convolution/kernel/ExponentialMovingAverage đź”´WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverageđź”´ WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage global_step

Is this

WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage

in checkpoint file the

WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage

that synthesize.py is looking for?

letmeetran commented 5 years ago

I am also receiving this error. Here is my terminal log:

root@floydhub:/floyd/home/Tacotron-2# python synthesize.py --model='WaveNet' --checkpoint='pretrained/wavenet_model.ckpt-17000' Using TensorFlow backend. Traceback (most recent call last): File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 73, in wavenet_synthesize checkpoint_path = tf.train.get_checkpoint_state(checkpoint).model_checkpoint_path AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "synthesize.py", line 100, in main() File "synthesize.py", line 92, in main wavenet_synthesize(args, hparams, wave_checkpoint) File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 76, in wavenet_synthesize raise RuntimeError('Failed to load checkpoint at {}'.format(checkpoint)) RuntimeError: Failed to load checkpoint at logs-WaveNet/wave_pretrained/wavenet_model.ckpt-17000 root@floydhub:/floyd/home/Tacotron-2# python synthesize.py --model='WaveNet' --checkpoint='trained/wavenet_model.ckpt-17000' Using TensorFlow backend. Traceback (most recent call last): File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 73, in wavenet_synthesize checkpoint_path = tf.train.get_checkpoint_state(checkpoint).model_checkpoint_path AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "synthesize.py", line 100, in main() File "synthesize.py", line 92, in main wavenet_synthesize(args, hparams, wave_checkpoint) File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 76, in wavenet_synthesize raise RuntimeError('Failed to load checkpoint at {}'.format(checkpoint)) RuntimeError: Failed to load checkpoint at logs-WaveNet/wave_trained/wavenet_model.ckpt-17000 root@floydhub:/floyd/home/Tacotron-2# python synthesize.py --model='WaveNet' --checkpoint='trained/wavenet_model.ckpt-17000' Using TensorFlow backend. Traceback (most recent call last): File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 74, in wavenet_synthesize checkpoint_path = tf.train.get_checkpoint_state(checkpoint).model_checkpoint_path AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "synthesize.py", line 100, in main() File "synthesize.py", line 92, in main wavenet_synthesize(args, hparams, wave_checkpoint) File "/floyd/home/Tacotron-2/wavenet_vocoder/synthesize.py", line 77, in wavenet_synthesize raise RuntimeError('Failed to load checkpoint at {}'.format(checkpoint)) RuntimeError: Failed to load checkpoint at logs-WaveNet/wave_trained/wavenet_model.ckpt-17000

I tried creating my own directory since the original directory (wave_pretrained/) did not work and hard-coded my new directory into wavenet_vocoder/synthesize.py but it still did not work.

Screen Shot 2019-08-01 at 3 49 21 PM Screen Shot 2019-08-01 at 3 49 37 PM
letmeetran commented 5 years ago

Update:

I am currently at the same stage as @JasonWei512. Below is the reason for this error:

Screen Shot 2019-08-01 at 6 31 06 PM Screen Shot 2019-08-01 at 6 30 55 PM

There is an extra block of information that our checkpoint does not write. I am currently trying to locate where this is occurring in the repo but I am having no luck. I have been using #64 as a reference. If anyone can point us in the right direction, that would be amazing.

JasonWei512 commented 5 years ago

I downgraded tensorflow from 1.14 to 1.10 as mentioned in #230 , and the error is gone

jieralice13 commented 5 years ago

@JasonWei512 I met the same issue when I run 'python synthesis.py --model 'WaveNet'. However, downgrading to 1.10 doesn't work for me. May I ask how you did the downgrading? I was using the provided Dockerfile, which had tensorflow-gpu=1.14.0 installed. Then I ran 'pip3 install tensorflow-gpu==1.10.0' to downgrade. However, the error is still there.

JasonWei512 commented 5 years ago

@JasonWei512 I met the same issue when I run 'python synthesis.py --model 'WaveNet'. However, downgrading to 1.10 doesn't work for me. May I ask how you did the downgrading? I was using the provided Dockerfile, which had tensorflow-gpu=1.14.0 installed. Then I ran 'pip3 install tensorflow-gpu==1.10.0' to downgrade. However, the error is still there.

I installed CUDA 9 and tensorflow 1.10, and retrained wavenet model.

jieralice13 commented 5 years ago

I see. I only installed cuda 9 and tf 1.10, but did not retrain wavenet. Thanks you for your help.

On Fri, Aug 16, 2019, 6:53 PM é­Źćť° notifications@github.com wrote:

@JasonWei512 https://github.com/JasonWei512 I met the same issue when I run 'python synthesis.py --model 'WaveNet'. However, downgrading to 1.10 doesn't work for me. May I ask how you did the downgrading? I was using the provided Dockerfile, which had tensorflow-gpu=1.14.0 installed. Then I ran 'pip3 install tensorflow-gpu==1.10.0' to downgrade. However, the error is still there.

I installed CUDA 9 and tensorflow 1.10, and retrained wavenet model.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Rayhane-mamah/Tacotron-2/issues/421?email_source=notifications&email_token=AETRSI2YSOS2OVXYLFU67TTQE5K3NA5CNFSM4IIQRWRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4QA5MA#issuecomment-522194608, or mute the thread https://github.com/notifications/unsubscribe-auth/AETRSIYRBIP34HUZI3NFRD3QE5K3NANCNFSM4IIQRWRA .

orascheg commented 4 years ago

Can it be that at saving of the checkpoints at WaveNet training the eval model (5,3M) of Wavenet is used instead the synth model (3,2M)? I am also experiencing this problem (tensorflow 1.14 GPU) and have put the following line into the function create_shadow_saver() in wavenet_vocoder/train.py log('Shadow variables {}'.format(shadow_variables)) When comparing the output of the shadow variables, I find that they are different if I start the training and the synthesis. Is it making sense to initialize the sh_saver directly after creating the training model (model, stats = model_train_mode(args, feeder, hparams, global_step)) and before the creation of the eval model (eval_model = model_test_mode(args, feeder, hparams, global_step))? I am not fully into the wirings of tensorflow, however I could imagine that the saver uses the latest model created. Perhaps this was different in tf 1.10

mib32 commented 4 years ago

@JasonWei512 can you please help, how to install cuda 9 if I already have 10?

JasonWei512 commented 4 years ago

@mib32 I uninstalled CUDA 10...