search

as-ideas / TransformerTTS

🤖💬 Transformer TTS: Implementation of a non-autoregressive Transformer based neural network for text to speech.
https://as-ideas.github.io/TransformerTTS/
Other
1.13k stars 227 forks source link

extract_durations.py slowly filling up RAM and gets killed #69

Closed padmalcom closed 4 years ago

padmalcom commented 4 years ago

Hi, I found time again to train a model on a larger German dataset. Creating the dataset works well but when I execute extract_durations.py the process is taking more and more RAM until the python process ends with a simple "killed". I have 32GB of RAM.

This is how I call it: (TransformerTTS) [user1@localhost TransformerTTS]$ python extract_durations.py --config config/melgan --binary --fix_jumps --fill_mode_next --store_predictions

These are the last three lines I see:

2020-10-26 08:43:29.819269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
Processing validation set: : 13it [00:10,  1.25it/s]
Processing training set: : 1444it [05:54,  1.38it/s]Getötet
cfrancesco commented 4 years ago

I set a large batch size in the script. You can try reducing that. I'll look into memory issues. Which repo version are you using? Does it include binned sampling already? This will reduce the memory consumption

padmalcom commented 4 years ago

Hi, just recognized that I had not updated my code for a while. I'll close this one and see if it works with the newest version.