search

CSTR-Edinburgh / merlin

This is now the official location of the Merlin project.
http://www.cstr.ed.ac.uk/projects/merlin/
Apache License 2.0
1.31k stars 441 forks source link

Problem during the acoustic training #360

Open nd39 opened 6 years ago

nd39 commented 6 years ago

Hi ,

I’m using the merlin toolkit with my own database (wav, labels and question file), something went rong when using the following configuration for acoustic training hybrid_rnn_2_tanh_1_lstm_1_gru. The result for this configuration is :

/home/ubuntu/anaconda3/lib/python3.6/site-packages/scipy/stats/stats.py:2998: RuntimeWarning: Mean of empty slice. mx = x.mean() /home/ubuntu/anaconda3/lib/python3.6/site-packages/scipy/stats/stats.py:2999: RuntimeWarning: Mean of empty slice. my = y.mean() 2018-05-12 11:35:44,041 INFO main : Develop: DNN -- MCD: 6.663 dB; BAP: 0.085 dB; F0:- RMSE: nan Hz; CORR: nan; VUV: 28.167% 2018-05-12 11:35:44,042 INFO main : Test : DNN -- MCD: 7.090 dB; BAP: 0.075 dB; F0:- RMSE: nan Hz; CORR: nan; VUV: 28.780%

I want to know why the RMSE is nan. Any idea please.

felipeespic commented 6 years ago

Hi, could you paste the output of the training step, please?

nd39 commented 6 years ago

The output of the training step is: 2018-05-12 11:08:47,206 INFO main: Installation information: 2018-05-12 11:08:47,206 INFO main: Merlin directory: /home/ubuntu/DNN/merlin-master 2018-05-12 11:08:47,206 INFO main: PATH: 2018-05-12 11:08:47,206 INFO main: /home/ubuntu/anaconda3/lib 2018-05-12 11:08:47,206 INFO main: /home/ubuntu/torch/install/bin 2018-05-12 11:08:47,206 INFO main: /home/ubuntu/torch/install/bin 2018-05-12 11:08:47,206 INFO main: /home/ubuntu/anaconda3/bin 2018-05-12 11:08:47,207 INFO main: /home/ubuntu/torch/install/bin 2018-05-12 11:08:47,207 INFO main: /home/ubuntu/torch/install/bin 2018-05-12 11:08:47,207 INFO main: /home/ubuntu/bin 2018-05-12 11:08:47,207 INFO main: /home/ubuntu/.local/bin 2018-05-12 11:08:47,207 INFO main: /usr/local/sbin 2018-05-12 11:08:47,207 INFO main: /usr/local/bin 2018-05-12 11:08:47,207 INFO main: /usr/sbin 2018-05-12 11:08:47,207 INFO main: /usr/bin 2018-05-12 11:08:47,207 INFO main: /sbin 2018-05-12 11:08:47,207 INFO main: /bin 2018-05-12 11:08:47,207 INFO main: /usr/games 2018-05-12 11:08:47,207 INFO main: /usr/local/games 2018-05-12 11:08:47,208 INFO main: /snap/bin 2018-05-12 11:08:47,208 INFO main: LD_LIBRARY_PATH: 2018-05-12 11:08:47,208 INFO main: /home/ubuntu/torch/install/lib 2018-05-12 11:08:47,208 INFO main: /home/ubuntu/torch/install/lib 2018-05-12 11:08:47,208 INFO main: /home/ubuntu/torch/install/lib 2018-05-12 11:08:47,208 INFO main: /home/ubuntu/torch/install/lib 2018-05-12 11:08:47,208 INFO main: Python version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) [GCC 7.2.0] 2018-05-12 11:08:47,208 INFO main: PYTHONPATH: 2018-05-12 11:08:47,208 INFO main: Numpy version: 1.14.0 2018-05-12 11:08:47,208 INFO main: Theano version: 1.0.1 2018-05-12 11:08:47,208 INFO main: THEANO_FLAGS: cuda.root=/usr/local/8.0,floatX=float32,on_unused_input=ignore 2018-05-12 11:08:47,208 INFO main: device: cpu 2018-05-12 11:08:47,280 INFO main: Execution information: 2018-05-12 11:08:47,318 INFO main: HOSTNAME: ubuntu-Aspire-ES1-571 2018-05-12 11:08:47,318 INFO main: USER: ubuntu 2018-05-12 11:08:47,319 INFO main: PID: 2364 2018-05-12 11:08:47,362 INFO main: Input label dimension is 69 2018-05-12 11:08:47,363 INFO main: preparing label data (input) using standard HTS style labels 2018-05-12 11:08:47,440 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0009.lab, 22 labels 2018-05-12 11:08:47,456 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0001.lab, 20 labels 2018-05-12 11:08:47,485 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0005.lab, 50 labels 2018-05-12 11:08:47,519 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0006.lab, 8 labels 2018-05-12 11:08:47,536 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0013.lab, 42 labels 2018-05-12 11:08:47,587 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0002.lab, 49 labels 2018-05-12 11:08:47,603 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0007.lab, 38 labels 2018-05-12 11:08:47,621 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0014.lab, 18 labels 2018-05-12 11:08:47,643 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0015.lab, 6 labels 2018-05-12 11:08:47,645 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0003.lab, 18 labels 2018-05-12 11:08:47,650 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0008.lab, 10 labels 2018-05-12 11:08:47,662 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0010.lab, 99 labels 2018-05-12 11:08:47,691 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0004.lab, 19 labels 2018-05-12 11:08:47,758 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0017.lab, 49 labels 2018-05-12 11:08:47,770 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0021.lab, 39 labels 2018-05-12 11:08:47,965 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0011.lab, 95 labels 2018-05-12 11:08:47,980 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0022.lab, 86 labels 2018-05-12 11:08:48,011 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0023.lab, 10 labels 2018-05-12 11:08:48,087 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0024.lab, 28 labels 2018-05-12 11:08:48,125 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0012.lab, 28 labels 2018-05-12 11:08:48,156 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0029.lab, 6 labels 2018-05-12 11:08:48,203 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0030.lab, 12 labels 2018-05-12 11:08:48,223 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0031.lab, 6 labels 2018-05-12 11:08:48,243 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0032.lab, 8 labels 2018-05-12 11:08:48,281 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0018.lab, 236 labels 2018-05-12 11:08:48,316 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0033.lab, 21 labels 2018-05-12 11:08:48,383 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0025.lab, 143 labels 2018-05-12 11:08:48,407 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0016.lab, 341 labels 2018-05-12 11:08:48,415 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0034.lab, 17 labels 2018-05-12 11:08:48,418 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0026.lab, 15 labels 2018-05-12 11:08:48,421 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0019.lab, 54 labels 2018-05-12 11:08:48,448 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0037.lab, 11 labels 2018-05-12 11:08:48,469 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0035.lab, 19 labels 2018-05-12 11:08:48,482 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0020.lab, 32 labels 2018-05-12 11:08:48,491 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0027.lab, 50 labels 2018-05-12 11:08:48,541 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0038.lab, 17 labels 2018-05-12 11:08:48,547 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0036.lab, 20 labels 2018-05-12 11:08:48,565 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0028.lab, 24 labels 2018-05-12 11:08:48,649 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0050.lab, 29 labels 2018-05-12 11:08:48,710 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0051.lab, 15 labels 2018-05-12 11:08:48,753 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0039.lab, 79 labels 2018-05-12 11:08:48,765 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0046.lab, 88 labels 2018-05-12 11:08:48,784 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0040.lab, 10 labels 2018-05-12 11:08:48,802 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0047.lab, 23 labels 2018-05-12 11:08:48,820 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0048.lab, 13 labels 2018-05-12 11:08:48,843 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0049.lab, 10 labels 2018-05-12 11:08:48,865 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0041.lab, 186 labels 2018-05-12 11:08:48,877 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0052.lab, 98 labels 2018-05-12 11:08:48,906 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0042.lab, 34 labels 2018-05-12 11:08:48,929 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0043.lab, 17 labels 2018-05-12 11:08:48,937 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0053.lab, 54 labels 2018-05-12 11:08:48,967 INFO labels: loaded /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/data/label_phone_align/cmu_us_arctic_slt_b0044.lab, 79 labels 2018-05-12 11:08:49,289 INFO acoustic_norm: across 48 files found min/max values of length 69: 2018-05-12 11:08:49,291 INFO acoustic_norm: min: [[0. 0. 0. 0. 0. 0.

          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.
          1. 0.04404987 0.45900714 0.04404987 2. ]] 2018-05-12 11:08:49,292 INFO acoustic_norm: max: [[ 1. 1. 1. 1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1.
          1. 1. 0.99733615 0.99733615 0.99733615 555. ]] 2018-05-12 11:08:49,470 INFO main: saved 69 vectors to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/label_norm_HTS_69.dat 2018-05-12 11:08:49,470 INFO main: creating acoustic (output) features 2018-05-12 11:08:49,574 INFO acoustic_comp: processing file 1 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0001.cmp 2018-05-12 11:08:49,575 INFO acoustic_comp: processing file 1 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0002.cmp 2018-05-12 11:08:49,575 INFO acoustic_comp: processing file 1 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0003.cmp 2018-05-12 11:08:49,575 INFO acoustic_comp: processing file 1 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0004.cmp 2018-05-12 11:08:52,931 INFO acoustic_comp: processing file 2 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0008.cmp 2018-05-12 11:08:52,933 INFO acoustic_comp: processing file 2 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0007.cmp 2018-05-12 11:08:53,119 INFO acoustic_comp: processing file 2 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0005.cmp 2018-05-12 11:08:54,620 INFO acoustic_comp: processing file 2 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0006.cmp 2018-05-12 11:08:54,905 INFO acoustic_comp: processing file 3 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0012.cmp 2018-05-12 11:08:56,085 INFO acoustic_comp: processing file 3 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0010.cmp 2018-05-12 11:08:58,066 INFO acoustic_comp: processing file 3 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0009.cmp 2018-05-12 11:08:58,645 INFO acoustic_comp: processing file 3 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0011.cmp 2018-05-12 11:08:58,680 INFO acoustic_comp: processing file 4 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0016.cmp 2018-05-12 11:09:00,973 INFO acoustic_comp: processing file 4 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0013.cmp 2018-05-12 11:09:04,516 INFO acoustic_comp: processing file 4 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0014.cmp 2018-05-12 11:09:06,142 INFO acoustic_comp: processing file 5 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0018.cmp 2018-05-12 11:09:07,859 INFO acoustic_comp: processing file 5 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0017.cmp 2018-05-12 11:09:10,225 INFO acoustic_comp: processing file 4 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0015.cmp 2018-05-12 11:09:11,348 INFO acoustic_comp: processing file 5 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0019.cmp 2018-05-12 11:09:12,315 INFO acoustic_comp: processing file 6 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0021.cmp 2018-05-12 11:09:15,544 INFO acoustic_comp: processing file 7 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0025.cmp 2018-05-12 11:09:16,141 INFO acoustic_comp: processing file 6 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0023.cmp 2018-05-12 11:09:18,206 INFO acoustic_comp: processing file 7 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0027.cmp 2018-05-12 11:09:24,736 INFO acoustic_comp: processing file 8 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0031.cmp 2018-05-12 11:09:26,056 INFO acoustic_comp: processing file 9 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0035.cmp 2018-05-12 11:09:28,659 INFO acoustic_comp: processing file 8 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0029.cmp 2018-05-12 11:09:29,476 INFO acoustic_comp: processing file 6 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0022.cmp 2018-05-12 11:09:29,527 INFO acoustic_comp: processing file 10 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0039.cmp 2018-05-12 11:09:29,662 INFO acoustic_comp: processing file 9 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0033.cmp 2018-05-12 11:09:34,295 INFO acoustic_comp: processing file 10 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0037.cmp 2018-05-12 11:09:36,741 INFO acoustic_comp: processing file 11 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0041.cmp 2018-05-12 11:09:39,001 INFO acoustic_comp: processing file 11 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0043.cmp 2018-05-12 11:09:39,223 INFO acoustic_comp: processing file 5 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0020.cmp 2018-05-12 11:09:40,373 INFO acoustic_comp: processing file 7 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0026.cmp 2018-05-12 11:09:40,763 INFO acoustic_comp: processing file 12 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0048.cmp 2018-05-12 11:09:41,643 INFO acoustic_comp: processing file 13 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0052.cmp 2018-05-12 11:09:42,381 INFO acoustic_comp: processing file 8 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0030.cmp 2018-05-12 11:09:42,761 INFO acoustic_comp: processing file 6 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0024.cmp 2018-05-12 11:09:45,779 INFO acoustic_comp: processing file 9 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0034.cmp 2018-05-12 11:09:46,942 INFO acoustic_comp: processing file 7 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0028.cmp 2018-05-12 11:09:50,432 INFO acoustic_comp: processing file 8 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0032.cmp 2018-05-12 11:09:51,163 INFO acoustic_comp: processing file 9 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0036.cmp 2018-05-12 11:09:51,830 INFO acoustic_comp: processing file 10 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0038.cmp 2018-05-12 11:09:52,683 INFO acoustic_comp: processing file 10 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0040.cmp 2018-05-12 11:09:53,167 INFO acoustic_comp: processing file 11 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0042.cmp 2018-05-12 11:09:53,325 INFO acoustic_comp: processing file 12 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0046.cmp 2018-05-12 11:09:53,509 INFO acoustic_comp: processing file 11 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0044.cmp 2018-05-12 11:09:56,721 INFO acoustic_comp: processing file 12 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0047.cmp 2018-05-12 11:09:58,020 INFO acoustic_comp: processing file 12 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0049.cmp 2018-05-12 11:09:58,322 INFO acoustic_comp: processing file 13 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0051.cmp 2018-05-12 11:09:58,360 INFO acoustic_comp: processing file 13 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0050.cmp 2018-05-12 11:09:58,731 INFO acoustic_comp: processing file 13 of 13 : /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_mgc_lf0_vuv_bap_187/cmu_us_arctic_slt_b0053.cmp 2018-05-12 11:10:02,238 INFO main: normalising acoustic (output) features using method MVN 2018-05-12 11:10:02,921 INFO main: saved MVN vectors to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/norm_info__mgc_lf0_vuv_bap_187_MVN.dat 2018-05-12 11:10:03,025 INFO main: saved mgc variance vector to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/var/mgc_180 2018-05-12 11:10:03,025 INFO main: saved lf0 variance vector to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/var/lf0_3 2018-05-12 11:10:03,025 INFO main: saved vuv variance vector to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/var/vuv_1 2018-05-12 11:10:03,026 INFO main: saved bap variance vector to /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/var/bap_3 2018-05-12 11:10:03,028 INFO main: label dimension is 69 2018-05-12 11:10:03,029 INFO main: training DNN 2018-05-12 11:10:03,260 INFO main.train_DNN: building the model 2018-05-12 11:10:54,461 INFO main.train_DNN: fine-tuning the DNN model 2018-05-12 11:12:40,040 INFO main.train_DNN: epoch 1, validation error 196.260559, train error 187.569550 time spent 105.58 2018-05-12 11:14:39,774 INFO main.train_DNN: epoch 2, validation error 196.226105, train error 187.439896 time spent 119.59 2018-05-12 11:16:19,797 INFO main.train_DNN: epoch 3, validation error 196.195877, train error 187.343918 time spent 99.89 2018-05-12 11:17:55,636 INFO main.train_DNN: epoch 4, validation error 196.168518, train error 187.270142 time spent 95.06 2018-05-12 11:19:30,020 INFO main.train_DNN: epoch 5, validation error 196.143204, train error 187.210007 time spent 94.25 2018-05-12 11:21:06,868 INFO main.train_DNN: epoch 6, validation error 196.119446, train error 187.158707 time spent 96.72 2018-05-12 11:22:41,170 INFO main.train_DNN: epoch 7, validation error 196.096924, train error 187.113342 time spent 94.16 2018-05-12 11:24:19,236 INFO main.train_DNN: epoch 8, validation error 196.075394, train error 187.072113 time spent 97.92 2018-05-12 11:25:54,942 INFO main.train_DNN: epoch 9, validation error 196.054703, train error 187.033966 time spent 95.58 2018-05-12 11:27:28,570 INFO main.train_DNN: epoch 10, validation error 196.034729, train error 186.998077 time spent 92.92 2018-05-12 11:29:02,024 INFO main.train_DNN: epoch 11, validation error 196.015350, train error 186.964005 time spent 92.86 2018-05-12 11:30:35,400 INFO main.train_DNN: epoch 12, validation error 195.996521, train error 186.931320 time spent 93.04 2018-05-12 11:32:07,757 INFO main.train_DNN: epoch 13, validation error 195.978180, train error 186.899811 time spent 92.20 2018-05-12 11:33:43,658 INFO main.train_DNN: epoch 14, validation error 195.960205, train error 186.869263 time spent 95.46 2018-05-12 11:35:23,982 INFO main.train_DNN: epoch 15, validation error 195.942627, train error 186.839569 time spent 99.80 2018-05-12 11:35:24,371 INFO main.train_DNN: overall training time: 24.50m validation error 195.942627 2018-05-12 11:35:24,376 INFO main: generating from DNN 2018-05-12 11:35:24,420 INFO dnn_generation: generating 1 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_no_silence_lab_norm_69/cmu_us_arctic_slt_b0050.lab 2018-05-12 11:35:27,855 INFO dnn_generation: generating 2 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_no_silence_lab_norm_69/cmu_us_arctic_slt_b0051.lab 2018-05-12 11:35:30,700 INFO dnn_generation: generating 3 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_no_silence_lab_norm_69/cmu_us_arctic_slt_b0052.lab 2018-05-12 11:35:34,260 INFO dnn_generation: generating 4 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/inter_module/nn_no_silence_lab_norm_69/cmu_us_arctic_slt_b0053.lab 2018-05-12 11:35:37,531 INFOparam_generation: processing 1 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/gen/hybrid_rnn_2_tanh_1_lstm_1_gru/cmu_us_arctic_slt_b0050.cmp 2018-05-12 11:35:37,641 INFOparam_generation: processing 2 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/gen/hybrid_rnn_2_tanh_1_lstm_1_gru/cmu_us_arctic_slt_b0051.cmp 2018-05-12 11:35:37,652 INFOparam_generation: processing 3 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/gen/hybrid_rnn_2_tanh_1_lstm_1_gru/cmu_us_arctic_slt_b0052.cmp 2018-05-12 11:35:37,681 INFOparam_generation: processing 4 of 4: /home/ubuntu/DNN/merlin-master/egs/build_your_own_voice/s1/experiments/my_voice/acoustic_model/gen/hybrid_rnn_2_tanh_1_lstm_1_gru/cmu_us_arctic_slt_b0053.cmp 2018-05-12 11:35:37,701 INFO main: reconstructing waveform(s) 2018-05-12 11:35:37,701 INFO wav_generation: creating waveform for 1 of 4: cmu_us_arctic_slt_b0050 2018-05-12 11:35:38,990 INFO wav_generation: creating waveform for 2 of 4: cmu_us_arctic_slt_b0051 2018-05-12 11:35:39,676 INFO wav_generation: creating waveform for 3 of 4: cmu_us_arctic_slt_b0052 2018-05-12 11:35:42,067 INFO wav_generation: creating waveform for 4 of 4: cmu_us_arctic_slt_b0053 2018-05-12 11:35:43,592 INFO main: calculating MCD 2018-05-12 11:35:44,041 INFO main: Develop: DNN -- MCD: 6.663 dB; BAP: 0.085 dB; F0:- RMSE: nan Hz; CORR: nan; VUV: 28.167% 2018-05-12 11:35:44,042 INFO main: Test : DNN -- MCD: 7.090 dB; BAP: 0.075 dB; F0:- RMSE: nan Hz; CORR: nan; VUV: 28.780%