search

barronalex / Tacotron

Implementation of Google's Tacotron in TensorFlow
236 stars 80 forks source link

High Memory Usage? #6

Closed zhang-jian closed 7 years ago

zhang-jian commented 7 years ago

Hi all,

Just wondering if anyone noticed that the memory usage is high?

My training data are: 925M Jun 16 08:09 mels.npy 56K Jun 16 08:09 meta.pkl 7.9K Jun 16 08:09 speech_lens.npy 21K Jun 16 13:45 stft_mean.npy 5.8G Jun 16 08:09 stfts.npy 21K Jun 16 13:45 stft_std.npy 7.9K Jun 16 08:09 text_lens.npy 771K Jun 16 08:09 texts.npy However, a "top" shows that training uses more than 20GB RES.

Thanks

Jian

j-pong commented 7 years ago

1%| | 11999/1000000 [3:05:46<284:20:45, 1.04s/it]saving weights WARNING:tensorflow:Error encountered when serializing LAYER_NAME_UIDS. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'dict' object has no attribute 'name' saving sample 1%|▏ | 13999/1000000 [3:46:16<355:12:20, 1.30s/it]saving weights WARNING:tensorflow:Error encountered when serializing LAYER_NAME_UIDS. Type is unsupported, or the types of the items don't match field type in CollectionDef. 'dict' object has no attribute 'name' saving sample 1%|▏ | 14993/1000000 [4:11:55<17855:26:07, 65.26s/it] Process finished with exit code 137 (interrupted by signal 9: SIGKILL)

Also i met problem with that ERROR

candlewill commented 7 years ago

Yes, I melt the same high usage of memory problem.

azraelkuan commented 7 years ago

yes, the same problem

barronalex commented 7 years ago

The latest commit should significantly improve this problem. I now store the spectrograms as float16s (this doesn't seem to affect sample quality) and create the numpy arrays in a more sensible way.

I found empirically that reading from memory speeded up training by about 2X over disk which hopefully is worth the higher memory usage.