cuDNN error: CUDNN_STATUS_BAD_PARAM

[luis-vera]

Hi. I tried training to tacotron (Windows 10 and one GPU GeForce RTX2080 Ti) and I got the following error message:

Thanks a lot for your help

python TTS/bin/train_tacotron.py --config_path TTS/tts/configs/config.json 2021-02-17 17:09:11.997602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll

Using CUDA: True Number of GPUs: 1 Mixed precision mode is ON Git Hash: e9e0784 Experiment folder: Models/LJSpeech/ljspeech-ddc-February-17-2021_05+09PM-e9e0784 Setting up Audio Processor... | > sample_rate:22050 | > resample:False | > num_mels:80 | > min_level_db:-100 | > frame_shift_ms:None | > frame_length_ms:None | > ref_level_db:20 | > fft_size:1024 | > power:1.5 | > preemphasis:0.0 | > griffin_lim_iters:60 | > signal_norm:True | > symmetric_norm:True | > mel_fmin:50.0 | > mel_fmax:7600.0 | > spec_gain:1.0 | > stft_pad_mode:reflect | > max_norm:4.0 | > clip_norm:True | > do_trim_silence:True | > trim_db:60 | > do_sound_norm:False | > stats_path:scale_stats.npy | > hop_length:256 | > win_length:1024 | > Found 13100 files in C:\Users\Voice-trainner\MozillaTTS\TTS\LJSpeech-1.1 Using model: Tacotron2

Model has 47914548 parameters

DataLoader initialization | > Use phonemes: True | > phoneme language: en-us | > Number of instances : 12969 | > Max length sequence: 187 | > Min length sequence: 5 | > Avg length sequence: 98.3403500655409 | > Num. instances discarded by max-min (max=153, min=6) seq limits: 476 | > Batch group size: 16.

EPOCH: 0/1000

Number of output frames: 7

TRAINING (2021-02-17 17:09:17) 2021-02-17 17:09:18.517650: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll Using CUDA: True Number of GPUs: 1 2021-02-17 17:09:21.598745: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll Using CUDA: True Number of GPUs: 1 2021-02-17 17:09:24.553649: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll Using CUDA: True Number of GPUs: 1 2021-02-17 17:09:27.454550: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll Using CUDA: True Number of GPUs: 1 ! Run is removed from Models/LJSpeech/ljspeech-ddc-February-17-2021_05+09PM-e9e0784 Traceback (most recent call last): File "TTS/bin/train_tacotron.py", line 721, in main(args) File "TTS/bin/train_tacotron.py", line 619, in main train_avg_loss_dict, global_step = train(train_loader, model, File "TTS/bin/train_tacotron.py", line 165, in train decoder_output, postnet_output, alignments, stop_tokens, decoder_backward_output, alignments_backward = model( File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, kwargs) File "c:\users\voice-trainner\proyecto\tts\TTS\tts\models\tacotron2.py", line 148, in forward encoder_outputs = self.encoder(embedded_inputs, text_lengths) File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\module.py", line 722, in _callimpl result = self.forward(*input, **kwargs) File "c:\users\voice-trainner\proyecto\tts\TTS\tts\layers\tacotron2.py", line 109, in forward o, = self.lstm(o) File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, kwargs) File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\rnn.py", line 579, in forward result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias, RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM

[erogol]

Not sure but looks like GPU OOM.

[erogol]

it is cross posting. I close this one.