stale[bot]
commented
3 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts
Hi everybody:
I am training MozillaTTS with Spanish Corpus (I am training on Windows 10) and I got this error message. I don't know if the error is because of GPU memory. THanks a lot for your help. Tracing is:
--> STEP: 594/709 -- GLOBAL_STEP: 260100 | > decoder_loss: 0.16054 (0.10841) | > postnet_loss: 0.13813 (0.09606) | > stopnet_loss: 0.03669 (0.16323) | > decoder_coarse_loss: 0.25214 (0.17344) | > decoder_ddc_loss: 0.00669 (0.01984) | > ga_loss: 0.00008 (0.00173) | > decoder_diff_spec_loss: 0.12294 (0.09193) | > postnet_diff_spec_loss: 0.11168 (0.08491) | > decoder_ssim_loss: 0.25527 (0.18237) | > postnet_ssim_loss: 0.23014 (0.16755) | > loss: 0.42462 (0.31522) | > align_error: 0.29491 (0.18657) | > max_spec_length: 466.0 | > max_text_length: 64.0 | > step_time: 0.7450 | > loader_time: 0.00 | > current_lr: 0.0001 ! Run is kept in Models/LJSpeech/ljspeech-ddc-May-17-2021_10+25AM-e9e0784 Traceback (most recent call last): File "TTS/bin/train_tacotron.py", line 721, in
main(args)
File "TTS/bin/train_tacotron.py", line 619, in main
train_avg_loss_dict, global_step = train(train_loader, model,
File "TTS/bin/train_tacotron.py", line 165, in train
decoder_output, postnet_output, alignments, stop_tokens, decoder_backward_output, alignments_backward = model(
File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
result = self.forward(*input, *kwargs)
File "c:\users\voice-trainner\proyecto\tts\TTS\tts\models\tacotron2.py", line 166, in forward
decoder_outputs, alignments, stop_tokens = self.decoder(
File "C:\Users\Voice-trainner\anaconda3\envs\TF2\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
result = self.forward(input, **kwargs)
File "c:\users\voice-trainner\proyecto\tts\TTS\tts\layers\tacotron2.py", line 327, in forward
decoder_output, attention_weights, stop_token = self.decode(memory)
File "c:\users\voice-trainner\proyecto\tts\TTS\tts\layers\tacotron2.py", line 267, in decode
query_input = torch.cat((memory, self.context), -1)
RuntimeError: CUDA error: unspecified launch failure