search

Rayhane-mamah / Tacotron-2

DeepMind's Tacotron-2 Tensorflow implementation
MIT License
2.28k stars 905 forks source link

Interrupting train.py #419

Closed nikigre closed 5 years ago

nikigre commented 5 years ago

Hi! I am running train.py on not enough powerful computer. And so I can't use the computer, while it is training. What would happen if I interrupt train process? Currently, I am on 1225 steps and to get to 5000 that is a long time. Can I later run train process again and it will continue where it left off? Thank you!

Ananas120 commented 5 years ago

Yes you can stop when you want and it will continue from last checkpoint

nikigre commented 5 years ago

Hi! Checkpoint was only made ad 300 steps. So I lost about 27 hours and 925 steps. Thank you for your answer.

Ananas120 commented 5 years ago

27 hours for only 975 steps is not logic or you run on CPU... in 27 hours i do around 64 000 steps on Colab ^^' You should run the code on colab, you can put all your code on github (or google drive) (and your dataset) and then run in Colab (with GPU environment) it takes around 2-4 sec / step for me (outputs_per_step equal 2 and max_mel_frames 900) You must also make checkpoint every 500 or 1000 steps and not close your colab tab (it closes your session after around 12hours if open and 1hour if closed) so you can load checkpoints (from colab) during training and re-import them when you continue training

nikigre commented 5 years ago

I have instaled TensorFlow-GPU but it always used CPU. And I was doing this on not soo powerful laptop. I just wanted to try if I would get speech to a level that is just a little understandable. And if that would be the case, I would use my server to do heavy work. I thought I modified parameters that would make checkpoint every 50 steps. But it wasn't... So I thought that code will automatically make a checkpoint after a keyboard interrupt. Besides speech quality in log directory is for a first 3 seconds a bit understandable but then it is just garbage...

Ananas120 commented 5 years ago

You can run it on Colab in one day you can have good voice (metallic but understandable)

nikigre commented 5 years ago

Oh really? I will try. Do you have any experience with other languages and this script? I want to try for Slovenian language...

Ananas120 commented 5 years ago

I used it with french dataset and the result is good but the voice is too metallic caused by the training data (i think the transformation (to linear / mel spectrogram) causes it but not sure

nikigre commented 5 years ago

Okay. Thank you for info. I am fairly new to this. Especially in Python programming language. Are there instructions on how to use Colab ant Tacotron-2?

Ananas120 commented 5 years ago

Ouf... if you are new it could be difficult but you can look tutorials i can’t explain it like this sorry ^^'

nikigre commented 5 years ago

No problem! I will eventually figure it out. ;) Thank you @Ananas120 for answers!

Ananas120 commented 5 years ago

No problem and good luck ! If you have precise question you can ask me ;)