search

srvk / eesen

The official repository of the Eesen project
http://arxiv.org/abs/1507.08240
Apache License 2.0
822 stars 342 forks source link

accuracy drops #122

Open madhavsund opened 7 years ago

madhavsund commented 7 years ago

I trained a model with 48 hrs of speech data and acheived a training accuracy of 95%. my network is 120 input neurons , 320X2 hidden layers and 67 output neurons.

but when I increased a speech corpus to 92 hrs of data and trained with the same network the accuracy drops after 9 th epoch...

I tried by increasing the hidden layers to 3 but the same problem

[NOTE] TOKEN_ACCURACY refers to token accuracy, i.e., (1.0 - token_error_rate). EPOCH 1 RUNNING ... ENDS [2017-Feb-1 11:50:23]: lrate 0.0001, TRAIN ACCURACY 48.5447%, VALID ACCURACY 19.5833% 19.5833 EPOCH 2 RUNNING ... ENDS [2017-Feb-1 15:07:33]: lrate 0.0001, TRAIN ACCURACY 63.9544%, VALID ACCURACY 27.6291% 8.0458 EPOCH 3 RUNNING ... ENDS [2017-Feb-1 19:18:59]: lrate 0.0001, TRAIN ACCURACY 68.8720%, VALID ACCURACY 28.3032% .6741 EPOCH 4 RUNNING ... ENDS [2017-Feb-1 22:49:47]: lrate 0.0001, TRAIN ACCURACY 69.9686%, VALID ACCURACY 35.5108% 7.2076 EPOCH 5 RUNNING ... ENDS [2017-Feb-2 00:59:58]: lrate 0.0001, TRAIN ACCURACY 72.8117%, VALID ACCURACY 29.4805% -6.0303 EPOCH 6 RUNNING ... ENDS [2017-Feb-2 03:09:58]: lrate 0.0001, TRAIN ACCURACY 72.5447%, VALID ACCURACY 26.6869% -2.7936 EPOCH 7 RUNNING ... ENDS [2017-Feb-2 05:20:23]: lrate 0.0001, TRAIN ACCURACY 73.6803%, VALID ACCURACY 28.0352% 1.3483 EPOCH 8 RUNNING ... ENDS [2017-Feb-2 07:30:45]: lrate 0.0001, TRAIN ACCURACY 73.9509%, VALID ACCURACY 30.8868% 2.8516 EPOCH 9 RUNNING ... ENDS [2017-Feb-2 09:41:22]: lrate 0.0001, TRAIN ACCURACY 71.7453%, VALID ACCURACY 22.9801% -7.9067 EPOCH 10 RUNNING ... ENDS [2017-Feb-2 11:54:40]: lrate 0.0001, TRAIN ACCURACY 68.3523%, VALID ACCURACY 22.7060% -.2741 EPOCH 11 RUNNING ... ENDS [2017-Feb-2 15:10:57]: lrate 0.0001, TRAIN ACCURACY 63.5763%, VALID ACCURACY 26.8709% 4.1649 EPOCH 12 RUNNING ... ENDS [2017-Feb-2 19:30:44]: lrate 0.0001, TRAIN ACCURACY 58.2389%, VALID ACCURACY 19.8774% -6.9935 EPOCH 13 RUNNING ... ENDS [2017-Feb-2 21:41:14]: lrate 0.0001, TRAIN ACCURACY 56.0561%, VALID ACCURACY 21.4257% 1.5483 EPOCH 14 RUNNING ... ENDS [2017-Feb-2 23:52:22]: lrate 0.0001, TRAIN ACCURACY 54.3079%, VALID ACCURACY 14.6172%

kindly help me to resolve this problem.

riebling commented 7 years ago

Aside from the obvious observation: something about the additional data is hurting, rather than helping (are the transcriptions accurate? Is the data noisy? were the features computed properly?) Usually decreasing accuracy with repeated epochs means it's time to start decreasing the learning rate. But that doesn't explain how the accuracies are so much lower with respect to the earlier, smaller data set.

It would help to see output from the same first 14 epochs of the experiment that gets 95% training accuracy. Is this newer experiment consistently more slow to improve than the earlier one? That would suggest (to me) the additional data is reducing performance "from the beginning"

Phrased differently: the accuracy shown here peaks at epoch 8. How well was the better-results experiment doing around epoch 8? And assuming all other things are equal. (It's less helpful to compare training accuracy history for completely different network dimesnionalities / topologies - this would be one of those "all other things" that should be kept equal across comparisons)

My colleagues might have better advice, but thanks for listening!

fmetze commented 7 years ago

Did you look at the training logs of the 9th iteration. Do you have overflows, NaNs, or simialr events?

madhavsund commented 7 years ago

thanks for your replies,

are the transcriptions accurate? yes, manually verified

Is the data noisy? no, recorded in room environment

were the features computed properly? how to check this

It would help to see output from the same first 14 epochs of the experiment that gets 95% training accuracy.

TRAINING STARTS [2016-Aug-1 14:27:24] [NOTE] TOKEN_ACCURACY refers to token accuracy, i.e., (1.0 - token_error_rate). EPOCH 1 RUNNING ... ENDS [2016-Aug-1 15:31:14]: lrate 0.0001, TRAIN ACCURACY 55.4951%, VALID ACCURACY 56.5166% 56.5166 EPOCH 2 RUNNING ... ENDS [2016-Aug-1 16:34:04]: lrate 0.0001, TRAIN ACCURACY 71.6535%, VALID ACCURACY 61.8603% 5.3437 EPOCH 3 RUNNING ... ENDS [2016-Aug-1 17:36:04]: lrate 0.0001, TRAIN ACCURACY 76.2801%, VALID ACCURACY 57.3913% -4.4690 EPOCH 4 RUNNING ... ENDS [2016-Aug-1 18:38:19]: lrate 0.0001, TRAIN ACCURACY 77.8907%, VALID ACCURACY 59.2572% 1.8659 EPOCH 5 RUNNING ... ENDS [2016-Aug-1 19:41:17]: lrate 0.0001, TRAIN ACCURACY 80.9326%, VALID ACCURACY 70.5037% 11.2465 EPOCH 6 RUNNING ... ENDS [2016-Aug-1 20:43:11]: lrate 0.0001, TRAIN ACCURACY 83.6738%, VALID ACCURACY 71.2827% .7790 EPOCH 7 RUNNING ... ENDS [2016-Aug-1 21:46:57]: lrate 0.0001, TRAIN ACCURACY 85.2399%, VALID ACCURACY 70.5964% -.6863 EPOCH 8 RUNNING ... ENDS [2016-Aug-1 22:49:30]: lrate 0.0001, TRAIN ACCURACY 86.6232%, VALID ACCURACY 72.4414% 1.8450 EPOCH 9 RUNNING ... ENDS [2016-Aug-1 23:51:59]: lrate 0.0001, TRAIN ACCURACY 87.9396%, VALID ACCURACY 71.3305% -1.1109 EPOCH 10 RUNNING ... ENDS [2016-Aug-2 00:54:33]: lrate 0.0001, TRAIN ACCURACY 88.9618%, VALID ACCURACY 74.6288% 3.2983 EPOCH 11 RUNNING ... ENDS [2016-Aug-2 01:56:34]: lrate 0.0001, TRAIN ACCURACY 89.4648%, VALID ACCURACY 74.8456% .2168 EPOCH 12 RUNNING ... ENDS [2016-Aug-2 02:59:15]: lrate 0.0001, TRAIN ACCURACY 90.1297%, VALID ACCURACY 75.3002% .4546 EPOCH 13 RUNNING ... ENDS [2016-Aug-2 04:01:34]: lrate 0.0001, TRAIN ACCURACY 90.8215%, VALID ACCURACY 75.9864% .6862 EPOCH 14 RUNNING ... ENDS [2016-Aug-2 05:03:47]: lrate 0.0001, TRAIN ACCURACY 91.5229%, VALID ACCURACY 75.9371% -.0493 EPOCH 15 RUNNING ... ENDS [2016-Aug-2 06:06:00]: lrate 5e-05, TRAIN ACCURACY 92.4408%, VALID ACCURACY 78.0527% 2.1156 EPOCH 16 RUNNING ... ENDS [2016-Aug-2 07:08:49]: lrate 2.5e-05, TRAIN ACCURACY 93.5363%, VALID ACCURACY 81.5185% 3.4658 EPOCH 17 RUNNING ... ENDS [2016-Aug-2 08:11:18]: lrate 1.25e-05, TRAIN ACCURACY 94.7391%, VALID ACCURACY 82.6712% 1.1527 EPOCH 18 RUNNING ... ENDS [2016-Aug-2 09:13:46]: lrate 6.25e-06, TRAIN ACCURACY 95.1161%, VALID ACCURACY 83.3665% .6953 EPOCH 19 RUNNING ... ENDS [2016-Aug-2 10:16:07]: lrate 3.125e-06, TRAIN ACCURACY 95.2809%, VALID ACCURACY 83.8659% .4994 EPOCH 20 RUNNING ... ENDS [2016-Aug-2 11:18:41]: lrate 1.5625e-06, TRAIN ACCURACY 95.3487%, VALID ACCURACY 84.3054% .4395 EPOCH 21 RUNNING ... ENDS [2016-Aug-2 12:21:28]: lrate 7.8125e-07, TRAIN ACCURACY 95.3613%, VALID ACCURACY 84.5342% .2288 EPOCH 22 RUNNING ... ENDS [2016-Aug-2 13:24:32]: lrate 3.90625e-07, TRAIN ACCURACY 95.3678%, VALID ACCURACY 84.7151% .1809 EPOCH 23 RUNNING ... ENDS [2016-Aug-2 14:27:07]: lrate 1.95313e-07, TRAIN ACCURACY 95.3934%, VALID ACCURACY 84.8287% .1136 EPOCH 24 RUNNING ... ENDS [2016-Aug-2 15:30:22]: lrate 9.76565e-08, TRAIN ACCURACY 95.4279%, VALID ACCURACY 84.8841% .0554 finished, too small rel. improvement .0554 Training succeeded. The final model exp/train_phn_l2_c320/final.nnet

Is this newer experiment consistently more slow to improve than the earlier one?
yes

Did you look at the training logs of the 9th iteration. Do you have overflows, NaNs, or simialr events? No such events

tr.iter9.txt

fmetze commented 7 years ago

All your weights are completely saturated in each LSTM layer, i.e. you have min=-50 and max=50 in the networks. You can reduce this with --max-grad (or so) when you create nnet.proto. I think some data is probably corrupt. Try training only on subsets, like you would do for cross-validation to see if you can out which parts may be corrupt?

On Feb 13, 2017, at 11:31 PM, shobana notifications@github.com wrote:

thanks for your replies,

are the transcriptions accurate? yes, manually verified

Is the data noisy? no, recorded in room environment

were the features computed properly? how to check this

It would help to see output from the same first 14 epochs of the experiment that gets 95% training accuracy.

TRAINING STARTS [2016-Aug-1 14:27:24] [NOTE] TOKEN_ACCURACY refers to token accuracy, i.e., (1.0 - token_error_rate). EPOCH 1 RUNNING ... ENDS [2016-Aug-1 15:31:14]: lrate 0.0001, TRAIN ACCURACY 55.4951%, VALID ACCURACY 56.5166% 56.5166 EPOCH 2 RUNNING ... ENDS [2016-Aug-1 16:34:04]: lrate 0.0001, TRAIN ACCURACY 71.6535%, VALID ACCURACY 61.8603% 5.3437 EPOCH 3 RUNNING ... ENDS [2016-Aug-1 17:36:04]: lrate 0.0001, TRAIN ACCURACY 76.2801%, VALID ACCURACY 57.3913% -4.4690 EPOCH 4 RUNNING ... ENDS [2016-Aug-1 18:38:19]: lrate 0.0001, TRAIN ACCURACY 77.8907%, VALID ACCURACY 59.2572% 1.8659 EPOCH 5 RUNNING ... ENDS [2016-Aug-1 19:41:17]: lrate 0.0001, TRAIN ACCURACY 80.9326%, VALID ACCURACY 70.5037% 11.2465 EPOCH 6 RUNNING ... ENDS [2016-Aug-1 20:43:11]: lrate 0.0001, TRAIN ACCURACY 83.6738%, VALID ACCURACY 71.2827% .7790 EPOCH 7 RUNNING ... ENDS [2016-Aug-1 21:46:57]: lrate 0.0001, TRAIN ACCURACY 85.2399%, VALID ACCURACY 70.5964% -.6863 EPOCH 8 RUNNING ... ENDS [2016-Aug-1 22:49:30]: lrate 0.0001, TRAIN ACCURACY 86.6232%, VALID ACCURACY 72.4414% 1.8450 EPOCH 9 RUNNING ... ENDS [2016-Aug-1 23:51:59]: lrate 0.0001, TRAIN ACCURACY 87.9396%, VALID ACCURACY 71.3305% -1.1109 EPOCH 10 RUNNING ... ENDS [2016-Aug-2 00:54:33]: lrate 0.0001, TRAIN ACCURACY 88.9618%, VALID ACCURACY 74.6288% 3.2983 EPOCH 11 RUNNING ... ENDS [2016-Aug-2 01:56:34]: lrate 0.0001, TRAIN ACCURACY 89.4648%, VALID ACCURACY 74.8456% .2168 EPOCH 12 RUNNING ... ENDS [2016-Aug-2 02:59:15]: lrate 0.0001, TRAIN ACCURACY 90.1297%, VALID ACCURACY 75.3002% .4546 EPOCH 13 RUNNING ... ENDS [2016-Aug-2 04:01:34]: lrate 0.0001, TRAIN ACCURACY 90.8215%, VALID ACCURACY 75.9864% .6862 EPOCH 14 RUNNING ... ENDS [2016-Aug-2 05:03:47]: lrate 0.0001, TRAIN ACCURACY 91.5229%, VALID ACCURACY 75.9371% -.0493 EPOCH 15 RUNNING ... ENDS [2016-Aug-2 06:06:00]: lrate 5e-05, TRAIN ACCURACY 92.4408%, VALID ACCURACY 78.0527% 2.1156 EPOCH 16 RUNNING ... ENDS [2016-Aug-2 07:08:49]: lrate 2.5e-05, TRAIN ACCURACY 93.5363%, VALID ACCURACY 81.5185% 3.4658 EPOCH 17 RUNNING ... ENDS [2016-Aug-2 08:11:18]: lrate 1.25e-05, TRAIN ACCURACY 94.7391%, VALID ACCURACY 82.6712% 1.1527 EPOCH 18 RUNNING ... ENDS [2016-Aug-2 09:13:46]: lrate 6.25e-06, TRAIN ACCURACY 95.1161%, VALID ACCURACY 83.3665% .6953 EPOCH 19 RUNNING ... ENDS [2016-Aug-2 10:16:07]: lrate 3.125e-06, TRAIN ACCURACY 95.2809%, VALID ACCURACY 83.8659% .4994 EPOCH 20 RUNNING ... ENDS [2016-Aug-2 11:18:41]: lrate 1.5625e-06, TRAIN ACCURACY 95.3487%, VALID ACCURACY 84.3054% .4395 EPOCH 21 RUNNING ... ENDS [2016-Aug-2 12:21:28]: lrate 7.8125e-07, TRAIN ACCURACY 95.3613%, VALID ACCURACY 84.5342% .2288 EPOCH 22 RUNNING ... ENDS [2016-Aug-2 13:24:32]: lrate 3.90625e-07, TRAIN ACCURACY 95.3678%, VALID ACCURACY 84.7151% .1809 EPOCH 23 RUNNING ... ENDS [2016-Aug-2 14:27:07]: lrate 1.95313e-07, TRAIN ACCURACY 95.3934%, VALID ACCURACY 84.8287% .1136 EPOCH 24 RUNNING ... ENDS [2016-Aug-2 15:30:22]: lrate 9.76565e-08, TRAIN ACCURACY 95.4279%, VALID ACCURACY 84.8841% .0554 finished, too small rel. improvement .0554 Training succeeded. The final model exp/train_phn_l2_c320/final.nnet

Is this newer experiment consistently more slow to improve than the earlier one? yes

Did you look at the training logs of the 9th iteration. Do you have overflows, NaNs, or simialr events? No such events

tr.iter9.txt https://github.com/srvk/eesen/files/773141/tr.iter9.txt — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/122#issuecomment-279605406, or mute the thread https://github.com/notifications/unsubscribe-auth/AEnA8cPmC0I12jBHjJ0dFIk28Zg3RmYBks5rcS4XgaJpZM4L_Emd.

Florian Metze http://www.cs.cmu.edu/directory/florian-metze Associate Research Professor Carnegie Mellon University