search

Rayhane-mamah / Tacotron-2

DeepMind's Tacotron-2 Tensorflow implementation
MIT License
2.27k stars 905 forks source link

Unable to synthesize using wavenet model. #230

Open ymitiku opened 6 years ago

ymitiku commented 6 years ago 
Traceback (most recent call last):
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key WaveNet_model/WaveNet_model/inference/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage not found in checkpoint
     [[Node: WaveNet_model/save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_WaveNet_model/save/Const_0_0, WaveNet_model/save/RestoreV2/tensor_names, WaveNet_model/save/RestoreV2/shape_and_slices)]]
     [[Node: WaveNet_model/save/RestoreV2/_231 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_236_WaveNet_model/save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "synthesize.py", line 99, in <module>
    main()
  File "synthesize.py", line 91, in main
    wavenet_synthesize(args, hparams, wave_checkpoint)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesize.py", line 78, in wavenet_synthesize
    run_synthesis(args, checkpoint_path, output_dir, hparams)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesize.py", line 19, in run_synthesis
    synth.load(checkpoint_path, hparams)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesizer.py", line 39, in load
    load_averaged_model(self.session, sh_saver, checkpoint_path)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/train.py", line 62, in load_averaged_model
    sh_saver.restore(sess, checkpoint_path)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key WaveNet_model/WaveNet_model/inference/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage not found in checkpoint
     [[Node: WaveNet_model/save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_WaveNet_model/save/Const_0_0, WaveNet_model/save/RestoreV2/tensor_names, WaveNet_model/save/RestoreV2/shape_and_slices)]]
     [[Node: WaveNet_model/save/RestoreV2/_231 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_236_WaveNet_model/save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'WaveNet_model/save/RestoreV2', defined at:
  File "synthesize.py", line 99, in <module>
    main()
  File "synthesize.py", line 91, in main
    wavenet_synthesize(args, hparams, wave_checkpoint)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesize.py", line 78, in wavenet_synthesize
    run_synthesis(args, checkpoint_path, output_dir, hparams)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesize.py", line 19, in run_synthesis
    synth.load(checkpoint_path, hparams)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/synthesizer.py", line 29, in load
    sh_saver = create_shadow_saver(self.model)
  File "/xx/xx/xx/Tacotron-2/wavenet_vocoder/train.py", line 59, in create_shadow_saver
    return tf.train.Saver(shadow_dict, max_to_keep=5)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
    build_save=build_save, build_restore=build_restore)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
    restore_sequentially, reshape)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 472, in _AddRestoreOps
    restore_sequentially)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/training/saver.py", line 886, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/xxx/xxxx/xxx/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key WaveNet_model/WaveNet_model/inference/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage not found in checkpoint
     [[Node: WaveNet_model/save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_WaveNet_model/save/Const_0_0, WaveNet_model/save/RestoreV2/tensor_names, WaveNet_model/save/RestoreV2/shape_and_slices)]]
     [[Node: WaveNet_model/save/RestoreV2/_231 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_236_WaveNet_model/save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

This is not the same as #163 because that issue is produced by running both models using python train.py --model ="Both". I've also checked the variables stored inside checkpoint file using tensorflows print_tensors_in_checkpoint_file function and it's output is:-

WaveNet_model/WaveNet_model/inference/conv1_d1x1/input_convolution/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/conv1_d1x1/input_convolution/kernel/ExponentialMovingAverage (DT_FLOAT) [1,1,64]
WaveNet_model/WaveNet_model/inference/conv1_d1x1_1/final_convolution_1/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/conv1_d1x1_1/final_convolution_1/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/conv1_d1x1_2/final_convolution_2/bias/ExponentialMovingAverage (DT_FLOAT) [2]
WaveNet_model/WaveNet_model/inference/conv1_d1x1_2/final_convolution_2/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,2]
WaveNet_model/WaveNet_model/inference/conv_transpose1d/bias/ExponentialMovingAverage (DT_FLOAT) [80]
WaveNet_model/WaveNet_model/inference/conv_transpose1d/kernel/ExponentialMovingAverage (DT_FLOAT) [1,10,80,80]
WaveNet_model/WaveNet_model/inference/conv_transpose1d_1/bias/ExponentialMovingAverage (DT_FLOAT) [80]
WaveNet_model/WaveNet_model/inference/conv_transpose1d_1/kernel/ExponentialMovingAverage (DT_FLOAT) [1,10,80,80]
WaveNet_model/WaveNet_model/inference/conv_transpose1d_2/bias/ExponentialMovingAverage (DT_FLOAT) [80]
WaveNet_model/WaveNet_model/inference/conv_transpose1d_2/kernel/ExponentialMovingAverage (DT_FLOAT) [1,22,80,80]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/causal_conv1d/residual_block_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/causal_conv1d/residual_block_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_1/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/causal_conv1d/residual_block_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/causal_conv1d/residual_block_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_10/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/causal_conv1d/residual_block_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/causal_conv1d/residual_block_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_11/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/causal_conv1d/residual_block_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/causal_conv1d/residual_block_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_12/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/causal_conv1d/residual_block_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/causal_conv1d/residual_block_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_13/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/causal_conv1d/residual_block_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/causal_conv1d/residual_block_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_14/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/causal_conv1d/residual_block_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/causal_conv1d/residual_block_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_15/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/causal_conv1d/residual_block_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/causal_conv1d/residual_block_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_16/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/causal_conv1d/residual_block_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/causal_conv1d/residual_block_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_17/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/causal_conv1d/residual_block_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/causal_conv1d/residual_block_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_18/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/causal_conv1d/residual_block_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/causal_conv1d/residual_block_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_19/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/causal_conv1d/residual_block_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/causal_conv1d/residual_block_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_2/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/causal_conv1d/residual_block_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/causal_conv1d/residual_block_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_3/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/causal_conv1d/residual_block_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/causal_conv1d/residual_block_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_4/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/causal_conv1d/residual_block_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/causal_conv1d/residual_block_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_5/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/causal_conv1d/residual_block_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/causal_conv1d/residual_block_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_6/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/causal_conv1d/residual_block_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/causal_conv1d/residual_block_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_7/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/causal_conv1d/residual_block_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/causal_conv1d/residual_block_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_8/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/causal_conv1d/residual_block_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/causal_conv1d/residual_block_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage (DT_FLOAT) [3,64,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage (DT_FLOAT) [128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1/residual_block_cin_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage (DT_FLOAT) [1,80,128]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1_1/residual_block_skip_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage (DT_FLOAT) [64]
WaveNet_model/WaveNet_model/inference/residual_conv1dglu_9/conv1_d1x1_2/residual_block_out_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage (DT_FLOAT) [1,64,64]
global_step (DT_INT32) []

as it can be seen from above log, there is no value for "WaveNet_model/WaveNet_model/inference/causal_conv1d/residual_block_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage" variable.

Rayhane-mamah commented 6 years ago

Hey @mitiku1 thanks for reaching out!

It's awesome that you made the checkpoint inspection as it is very helpful. As one can see, it is clear that the residual_conv1dglu/ component is being added to the scopes during training but not during tests which is causing this whole issue. I have had this issue a couple of times during development and I didn't really know how it went away. So please, consider these two scenarios:

  1. Did you train your wavenet model with a previous repo version then updated the repo when trying synthesis?
  2. If not, could you please provide steps to reproduce?

As for now, for a quick temporary solution (that does not require training), you can rename your checkpoint scopes during synthesis by taking off the residual_conv1dglu/ component to ensure the synthesis graph is consistent with the training one.

Thank you for reporting this bug, you answer to my above question will help us solve it for the long run.

ymitiku commented 6 years ago

@Rayhane-mamah Thanks for your replay.To answer your question

  1. I've trained the wavenet model and try to synthesize from the same version of the repo
  2. I followed the following steps to reproduce the issue. To reproduce the issue I've made changes before running training program.
    • It should be noted that when I clone the repository, head commit of master branch was @this commit
    • I am using tensorflow version 1.8.0
    • I've placed LJ dataset inside project folder
    • I've created test.txt file and placed a single sentence("Hello world!"), for evaluation.

STEP - 1 Makeing some changes.

I will explain why I made each change.

Change-1: I've made some changes to hparam.py (To deal with low memory gpu). If you want to see the changes I may post it here. Change-2 : I've changed the --tacotron_train_steps`` and ``--wavenet_train_steps values of train.py to 200, to reproduce the issue quickly. Change-3: I was getting AttributeError: 'tuple' object has no attribute '_uses_learning_phase' error message for calling functions which return tuple. So I changed tuple return values of some functions [ResidualConv1DGLU.call,CausalConv1D.call] to lists. Change-4: After completing this I was getting another error message. TypeError: This Layer takes aninputsargument to call(), and only theinputsargument may be specified as a positional argument. Pass everything else as a keyword argument (those arguments will not be tracked as inputs to the Layer).. To fix this I've changed some lines to call methods with keyword arguments.[ResidualConv1DGLU.incremental_step,WaveNet.step] Change 5: I've replaced info argument name of plot.plot_alignment method call with title argument [tacotron.Synthesizer.synthesize(1,2,3)] (This is because the method definition does not have info parameter. Change 6: In wavenet_vocoder.modules.CausalConv1D.call method when convolution_queue is None, I was getting error message AttributeError: 'NoneType' object has no attribute '_keras_mask'. To fix this issue I've changed the following code.

                        return tf.reshape(output, [batch_size, 1, self.layer.filters]), convolution_queue

to 

                        if convolution_queue is None:
                    return tf.reshape(output, [batch_size, 1, self.layer.filters])
            else:
                    return [tf.reshape(output, [batch_size, 1, self.layer.filters]), convolution_queue]

STEP - 2 running preprocessor.py

python preprocess.py

STEP - 3 Training Tacotron model

python train.py --model "Tacotron"

STEP - 4 Synthesis training data for wavenet using Tacotron model

python synthesize.py --model Tacotron --GTA True --mode synthesis

STEP - 5 Training WaveNet model

python train.py --model "WaveNet"

STEP-6 Synthesizing mels with Tacotron for evaluation

python synthesize.py --model Tacotron --GTA False --mode eval --text_list test.txt

STEP-7 Synthsizing audio with wavenet model.

python synthesize.py --model WaveNet --mode eval

After this the program throws the error message.

Rayhane-mamah commented 6 years ago

Hello again @mitiku1, first of all thanks for all the details.

So considering I have moved from the previous commit a while back, I was not able to reproduce locally, I must have fixed it sometime between the previous and the current version. In any case, please try updating to latest version and let me know if the problem persists!

EDIT: keras wrappers bugs are still existent for tf 1.8 or later users. Working on an efficient fix. but your workarounds can work for now EDIT 2: I have pushed another commit with your fix, let me know if I missed anything! Thanks for the contribution :)

ymitiku commented 6 years ago

@Rayhane-mamah thanks again. Though I've tried with latest version of the repo the error message was persisted, till I completely changed the virtual environment. I installed tensorflow 1.10.0 and other libraries from requirement file of this repository. This way now the error message has gone. I'm able to run synthesize.py for wavenet vocoder. I'm not sure if changing the tensorflow version or changing version of other libraries solved the problem.

One last thing, though this is not the case for tensorflow 1.10, I think you forget here to pass values for non input argument as positional arguments.

bhavuk0909 commented 4 years ago

Can anyone tell me how to use synthesize.py file on the pretrained model?