Error while training wavenet

[ChenXi1992]

The error occurred in training WaveNet, I tried the different tensorflow-gpu version, it doesn't work.

I didn't change any code, and only install tensorflow-gpu All the information in Google doesn't help.

Thanks in advance

Error info:

---

Using TensorFlow backend. Checkpoint_path: logs-WaveNet/wave_pretrained/wavenet_model.ckpt Loading training data from: tacotron_output/gta/map.txt Using model: WaveNet Hyperparameters: allow_clipping_in_normalization: True attention_dim: 128 attention_filters: 32 attention_kernel: (31,) cin_channels: 80 cleaners: english_cleaners clip_mels_length: True cross_entropy_pos_weight: 20 cumulative_weights: True decoder_layers: 2 decoder_lstm_units: 1024 embedding_dim: 512 enc_conv_channels: 512 enc_conv_kernel_size: (5,) enc_conv_num_layers: 3 encoder_lstm_units: 256 fmax: 7600 fmin: 25 frame_shift_ms: None freq_axis_kernel_size: 3 gate_channels: 512 gin_channels: -1 griffin_lim_iters: 60 hop_size: 256 input_type: raw kernel_size: 3 layers: 24 log_scale_min: -32.23619130191664 mask_decoder: True mask_encoder: True max_abs_value: 4.0 max_iters: 2500 max_mel_frames: 900 max_time_sec: None max_time_steps: 13000 min_level_db: -100 n_fft: 1024 natural_eval: False num_freq: 513 num_gpus: 1 num_mels: 80 out_channels: 30 outputs_per_step: 2 postnet_channels: 512 postnet_kernel_size: (5,) postnet_num_layers: 5 power: 1.2 predict_linear: False prenet_layers: [256, 256] quantize_channels: 65536 ref_level_db: 20 rescale: True rescaling_max: 0.999 residual_channels: 512 sample_rate: 22050 signal_normalization: True silence_threshold: 2 skip_out_channels: 256 smoothing: False stacks: 4 stop_at_any: True symmetric_mels: True tacotron_adam_beta1: 0.9 tacotron_adam_beta2: 0.999 tacotron_adam_epsilon: 1e-06 tacotron_batch_size: 48 tacotron_data_random_state: 1234 tacotron_decay_learning_rate: True tacotron_decay_rate: 0.2 tacotron_decay_steps: 40000 tacotron_dropout_rate: 0.5 tacotron_final_learning_rate: 1e-05 tacotron_initial_learning_rate: 0.001 tacotron_random_seed: 5339 tacotron_reg_weight: 1e-06 tacotron_scale_regularization: True tacotron_start_decay: 50000 tacotron_swap_with_cpu: False tacotron_teacher_forcing_decay_alpha: 0.0 tacotron_teacher_forcing_decay_steps: 280000 tacotron_teacher_forcing_final_ratio: 0.0 tacotron_teacher_forcing_init_ratio: 1.0 tacotron_teacher_forcing_mode: constant tacotron_teacher_forcing_ratio: 1.0 tacotron_teacher_forcing_start_decay: 10000 tacotron_test_batches: 48 tacotron_test_size: None tacotron_zoneout_rate: 0.1 train_with_GTA: False trim_fft_size: 512 trim_hop_size: 128 trim_silence: True trim_top_db: 60 upsample_conditional_features: True upsample_scales: [16, 16] use_all_gpus: False use_bias: True use_lws: True wavenet_adam_beta1: 0.9 wavenet_adam_beta2: 0.999 wavenet_adam_epsilon: 1e-06 wavenet_batch_size: 4 wavenet_data_random_state: 1234 wavenet_dropout: 0.05 wavenet_ema_decay: 0.9999 wavenet_learning_rate: 0.0001 wavenet_random_seed: 5339 wavenet_swap_with_cpu: False wavenet_test_batches: None wavenet_test_size: 0.0441 win_size: None Initializing Wavenet model. Dimensions (? = dynamic shape): Train mode: True Eval mode: False Synthesis mode: False inputs: (?, 1, ?) local_condition: (?, 80, ?) targets: (?, ?) outputs: (?, ?) Initializing Wavenet model. Dimensions (? = dynamic shape): Train mode: False Eval mode: True Synthesis mode: False local_condition: (1, 80, ?) targets: (?,) outputs: (?,) Wavenet training set to a maximum of 360000 steps No model to load at logs-WaveNet/wave_pretrained/

Generated 32 train batches of size 4 in 0.285 sec

Generated 578 test batches of size 1 in 0.906 sec

Exiting due to Exception: Conv2DCustomBackpropInputOp only supports NHWC. [[Node: model/inference/conv2d_transpose/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 16], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model/inference/conv2d_transpose/stack, model/inference/conv2d_transpose/kernel/read, model/inference/ExpandDims_1)]]

Caused by op 'model/inference/conv2d_transpose/conv2d_transpose', defined at: File "train.py", line 127, in main() File "train.py", line 119, in main wavenet_train(args, log_dir, hparams, args.wavenet_input) File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/train.py", line 244, in wavenet_train return train(log_dir, args, hparams, input_path) File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/train.py", line 167, in train model, stats = model_train_mode(args, feeder, hparams, global_step) File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/train.py", line 117, in model_train_mode feeder.input_lengths, x=feeder.inputs) File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/models/wavenet.py", line 169, in initialize y_hat = self.step(x, c, g, softmax=False) #softmax is automatically computed inside softmax_cross_entropy if needed File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/models/wavenet.py", line 435, in step c = transposed_conv(c) File "/home/rick/Documents/TTS-Xi/Tacotron-2-master/wavenet_vocoder/models/modules.py", line 333, in call return self.convt(inputs) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 717, in call outputs = self.call(inputs, *args, **kwargs) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1667, in call data_format=utils.convert_data_format(self.data_format, ndim=4)) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1254, in conv2d_transpose name=name) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1228, in conv2d_backprop_input dilations=dilations, name=name) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op op_def=op_def) File "/home/rick/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC. [[Node: model/inference/conv2d_transpose/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 16], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model/inference/conv2d_transpose/stack, model/inference/conv2d_transpose/kernel/read, model/inference/ExpandDims_1)]]

rick@rick-PowerEdge-R720:~/Documents/TTS-Xi/Tacotron-2-master$ python3 train.py --model='WaveNet'

[ben-8878]

@ChenXi1992 I met same error, how do you solve it?

[ChenXi1992]

@v-yunbin Hey, there is another same Issue which is still open. I posted my solution in there, you can look it up.