Erro in train Tacotron without Wavenet

I'm getting this error, when I train tacotron

I already placed tacotron batch_size in 4, but still

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue. Using TensorFlow backend. Checkpoint path: logs-Tacotron/taco_pretrained/tacotron_model.ckpt Loading training data from: ../locutores/training_data/train.txt Using model: Tacotron Hyperparameters: GL_on_GPU: False NN_init: True NN_scaler: 0.3 allow_clipping_in_normalization: False attention_dim: 128 attention_filters: 32 attention_kernel: (31,) attention_win_size: 7 batch_norm_position: after cbhg_conv_channels: 128 cbhg_highway_units: 128 cbhg_highwaynet_layers: 4 cbhg_kernels: 8 cbhg_pool_size: 2 cbhg_projection: 256 cbhg_projection_kernel_size: 3 cbhg_rnn_units: 128 cdf_loss: False cin_channels: 80 cleaners: basic_cleaners clip_for_wavenet: False clip_mels_length: False clip_outputs: False cross_entropy_pos_weight: 1 cumulative_weights: True decoder_layers: 2 decoder_lstm_units: 1024 embedding_dim: 512 enc_conv_channels: 512 enc_conv_kernel_size: (5,) enc_conv_num_layers: 3 encoder_lstm_units: 256 fmax: 7600 fmin: 55 frame_shift_ms: None freq_axis_kernel_size: 3 gate_channels: 256 gin_channels: -1 griffin_lim_iters: 60 hop_size: 1600 input_type: raw kernel_size: 3 layers: 20 leaky_alpha: 0.4 legacy: True log_scale_min: -32.23619130191664 log_scale_min_gauss: -16.11809565095832 lower_bound_decay: 0.1 magnitude_power: 2.0 mask_decoder: False mask_encoder: True max_abs_value: 4.0 max_iters: 10000 max_mel_frames: 900 max_time_sec: None max_time_steps: 11000 min_level_db: -100 n_fft: 1024 n_speakers: 5 normalize_for_wavenet: False num_freq: 513 num_mels: 20 out_channels: 2 outputs_per_step: 1 postnet_channels: 512 postnet_kernel_size: (5,) postnet_num_layers: 5 power: 1.5 predict_linear: False preemphasis: 0.97 preemphasize: False prenet_layers: [256, 256] quantize_channels: 65536 ref_level_db: 20 rescale: False rescaling_max: 0.999 residual_channels: 128 residual_legacy: True sample_rate: 16000 signal_normalization: False silence_threshold: 2 skip_out_channels: 128 smoothing: False speakers: ['speaker0', 'speaker1', 'speaker2', 'speaker3', 'speaker4'] speakers_path: None split_on_cpu: True stacks: 2 stop_at_any: True symmetric_mels: False synthesis_constraint: False synthesis_constraint_type: window tacotron_adam_beta1: 0.9 tacotron_adam_beta2: 0.999 tacotron_adam_epsilon: 1e-06 tacotron_batch_size: 32 tacotron_clip_gradients: True tacotron_data_random_state: 1234 tacotron_decay_learning_rate: True tacotron_decay_rate: 0.5 tacotron_decay_steps: 18000 tacotron_dropout_rate: 0.5 tacotron_final_learning_rate: 1e-05 tacotron_fine_tuning: False tacotron_initial_learning_rate: 0.001 tacotron_natural_eval: False tacotron_num_gpus: 1 tacotron_random_seed: 5339 tacotron_reg_weight: 1e-06 tacotron_scale_regularization: False tacotron_start_decay: 30000 tacotron_swap_with_cpu: False tacotron_synthesis_batch_size: 1 tacotron_teacher_forcing_decay_alpha: None tacotron_teacher_forcing_decay_steps: 40000 tacotron_teacher_forcing_final_ratio: 0.0 tacotron_teacher_forcing_init_ratio: 1.0 tacotron_teacher_forcing_mode: constant tacotron_teacher_forcing_ratio: 1.0 tacotron_teacher_forcing_start_decay: 10000 tacotron_test_batches: 1 tacotron_test_size: 0.05 tacotron_zoneout_rate: 0.1 train_with_GTA: False trim_fft_size: 2048 trim_hop_size: 512 trim_silence: False trim_top_db: 40 upsample_activation: Relu upsample_scales: [11, 25] upsample_type: SubPixel use_bias: True use_lws: False use_speaker_embedding: False wavenet_adam_beta1: 0.9 wavenet_adam_beta2: 0.999 wavenet_adam_epsilon: 1e-06 wavenet_batch_size: 8 wavenet_clip_gradients: True wavenet_data_random_state: 1234 wavenet_debug_mels: ['training_data/mels/mel-LJ001-0008.npy'] wavenet_debug_wavs: ['training_data/audio/audio-LJ001-0008.npy'] wavenet_decay_rate: 0.5 wavenet_decay_steps: 200000 wavenet_dropout: 0.05 wavenet_ema_decay: 0.9999 wavenet_gradient_max_norm: 100.0 wavenet_gradient_max_value: 5.0 wavenet_init_scale: 1.0 wavenet_learning_rate: 0.001 wavenet_lr_schedule: exponential wavenet_natural_eval: False wavenet_num_gpus: 1 wavenet_pad_sides: 1 wavenet_random_seed: 5339 wavenet_swap_with_cpu: False wavenet_synth_debug: False wavenet_synthesis_batch_size: 20 wavenet_test_batches: 1 wavenet_test_size: None wavenet_warmup: 4000.0 wavenet_weight_normalization: False win_size: 800 Loaded metadata for 95849 examples (128.90 hours) WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py:64: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, use tf.py_function, which takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means tf.py_functions can use accelerators such as GPUs as well as being differentiable using a gradient tape.

WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:100: LSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version. Instructions for updating: This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:387: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv1d instead. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:388: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.batch_normalization instead. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:391: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:215: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.Bidirectional(keras.layers.RNN(cell)), which is equivalent to this API WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.RNN(cell), which is equivalent to this API WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:279: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version. Instructions for updating: This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0. WARNING:tensorflow:From /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:246: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. initialisation done /gpu:0 Initialized Tacotron model. Dimensions (? = dynamic shape): Train mode: True Eval mode: False GTA mode: False Synthesis mode: False Input: (?, ?) device: 0 embedding: (?, ?, 512) enc conv out: (?, ?, 512) encoder out: (?, ?, 512) decoder out: (?, ?, 20) residual out: (?, ?, 512) projected residual out: (?, ?, 20) mel out: (?, ?, 20)

out: (?, ?) Tacotron Parameters 26.891 Million. WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:667: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. initialisation done /gpu:0 Initialized Tacotron model. Dimensions (? = dynamic shape): Train mode: False Eval mode: True GTA mode: False Synthesis mode: False Input: (?, ?) device: 0 embedding: (?, ?, 512) enc conv out: (?, ?, 512) encoder out: (?, ?, 512) decoder out: (?, ?, 20) residual out: (?, ?, 512) projected residual out: (?, ?, 20) mel out: (?, ?, 20) out: (?, ?) Tacotron Parameters 26.891 Million. Tacotron training set to a maximum of 200000 steps Loading checkpoint logs-Tacotron/taco_pretrained/tacotron_model.ckpt-0 WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. Generated 64 train batches of size 32 in 2.408 sec Generated 149 test batches of size 32 in 3.775 sec Step 1 [10.147 sec/step, loss=14.91386, avg_loss=14.91386] Saving Model Character Embeddings visualization.. Tacotron Character embeddings have been updated on tensorboard! Exiting due to exception: OOM when allocating tensor with shape[32,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:134) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[node Tacotron_model/clip_by_global_norm/mul_38 (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py:429) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. Caused by op 'Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform', defined at: File "train.py", line 138, in main() File "train.py", line 128, in main tacotron_train(args, log_dir, hparams) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 399, in tacotron_train return train(log_dir, args, hparams) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 156, in train model, stats = model_train_mode(args, feeder, hparams, global_step) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 91, in model_train_mode is_training=True, split_infos=feeder.split_infos) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py", line 173, in initialize swap_memory=hp.tacotron_swap_with_cpu) File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 322, in dynamic_decode swap_memory=swap_memory) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop return_same_structure) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop body_result = body(*packed_vars_for_body) File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 265, in body decoder_finished) = decoder.step(time, inputs, state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/custom_decoder.py", line 118, in step (cell_outputs, stop_token), cell_state = self._cell(inputs, state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/Architecture_wrappers.py", line 177, in __call__ LSTM_output, next_cell_state = self._cell(LSTM_input, state.cell_state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py", line 283, in __call__ return self._cell(inputs, states) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 234, in __call__ return super(RNNCell, self).__call__(inputs, state) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/layers/base.py", line 530, in __call__ outputs = super(Layer, self).__call__(inputs, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in __call__ outputs = self.call(inputs, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1515, in call cur_inp, new_state = cell(cur_inp, cur_state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py", line 134, in __call__ h = (1 - self._zoneout_outputs) * tf.nn.dropout(new_h - prev_h, (1 - self._zoneout_outputs)) + prev_h File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 2979, in dropout return dropout_v2(x, rate, noise_shape=noise_shape, seed=seed, name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 3048, in dropout_v2 noise_shape, seed=seed, dtype=x.dtype) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_random_ops.py", line 777, in random_uniform name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:134) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[node Tacotron_model/clip_by_global_norm/mul_38 (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py:429) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[{{node Tacotron_model/clip_by_global_norm/mul_38}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 225, in train step, loss, opt = sess.run([global_step, model.loss, model.optimize]) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:134) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[node Tacotron_model/clip_by_global_norm/mul_38 (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py:429) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. Caused by op 'Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform', defined at: File "train.py", line 138, in main() File "train.py", line 128, in main tacotron_train(args, log_dir, hparams) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 399, in tacotron_train return train(log_dir, args, hparams) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 156, in train model, stats = model_train_mode(args, feeder, hparams, global_step) File "/home/manuel_garcia02/Tacotron-2/tacotron/train.py", line 91, in model_train_mode is_training=True, split_infos=feeder.split_infos) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py", line 173, in initialize swap_memory=hp.tacotron_swap_with_cpu) File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 322, in dynamic_decode swap_memory=swap_memory) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop return_same_structure) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop body_result = body(*packed_vars_for_body) File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/seq2seq/python/ops/decoder.py", line 265, in body decoder_finished) = decoder.step(time, inputs, state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/custom_decoder.py", line 118, in step (cell_outputs, stop_token), cell_state = self._cell(inputs, state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/Architecture_wrappers.py", line 177, in __call__ LSTM_output, next_cell_state = self._cell(LSTM_input, state.cell_state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py", line 283, in __call__ return self._cell(inputs, states) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 234, in __call__ return super(RNNCell, self).__call__(inputs, state) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/layers/base.py", line 530, in __call__ outputs = super(Layer, self).__call__(inputs, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in __call__ outputs = self.call(inputs, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1515, in call cur_inp, new_state = cell(cur_inp, cur_state) File "/home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py", line 134, in __call__ h = (1 - self._zoneout_outputs) * tf.nn.dropout(new_h - prev_h, (1 - self._zoneout_outputs)) + prev_h File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 2979, in dropout return dropout_v2(x, rate, noise_shape=noise_shape, seed=seed, name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 3048, in dropout_v2 noise_shape, seed=seed, dtype=x.dtype) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_random_ops.py", line 777, in random_uniform name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_LSTM/decoder_LSTM/multi_rnn_cell/cell_1/dropout_1/random_uniform/RandomUniform (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/modules.py:134) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[node Tacotron_model/clip_by_global_norm/mul_38 (defined at /home/manuel_garcia02/Tacotron-2/tacotron/models/tacotron.py:429) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Rayhane-mamah / Tacotron-2

Erro in train Tacotron without Wavenet #432