search

cmusphinx / g2p-seq2seq

G2P with Tensorflow
Other
667 stars 196 forks source link

Handling of stopping criteria #140

Open ellurunaresh opened 6 years ago

ellurunaresh commented 6 years ago

I am training g2p model. The loss between step 191401 and 191501 was very small ideally it should stop the training but didn't happen. Is there any way to set the stopping criteria during training the model. Please give me guidance.

INFO:tensorflow:step = 191201, loss = 0.00049841707 (75.472 sec) [2018-07-05 14:09:17,385] step = 191201, loss = 0.00049841707 (75.472 sec) INFO:tensorflow:global_step/sec: 1.32075 [2018-07-05 14:10:33,098] global_step/sec: 1.32075 INFO:tensorflow:step = 191301, loss = 0.00062228605 (75.715 sec) [2018-07-05 14:10:33,099] step = 191301, loss = 0.00062228605 (75.715 sec) INFO:tensorflow:global_step/sec: 1.3279 [2018-07-05 14:11:48,405] global_step/sec: 1.3279 INFO:tensorflow:step = 191401, loss = 0.00045523737 (75.307 sec) [2018-07-05 14:11:48,407] step = 191401, loss = 0.00045523737 (75.307 sec) INFO:tensorflow:global_step/sec: 1.32259 [2018-07-05 14:13:04,014] global_step/sec: 1.32259 INFO:tensorflow:step = 191501, loss = 0.00040539462 (75.609 sec) [2018-07-05 14:13:04,015] step = 191501, loss = 0.00040539462 (75.609 sec)

nurtas-m commented 6 years ago

Hello, @ellurunaresh No, we did not include the stopping criteria still.

ellurunaresh commented 6 years ago

Thanks for the reply. 1)Is there any documentation or blog which could explain the training log details. 2) Can I change the learning rate and optimizer function?

nurtas-m commented 6 years ago

1) All of the training log details printed by tensor2tensor's transformer model. So, I think you may search the explanations of the logs in tensor2tensor project. 2) Please, clone the latest version of g2p-seq2seq. I have just added the possibility of changing hparams. So, if you want to change learning_rate, optimizer or other parameters defined in transformer_base hyper parameters set, you can just specify them in --hparams flag: g2p-seq2seq --train train_dictionary.dic --model_dir model_folder_path --hparams "learning_rate=0.5,optimizer=Adam" All default values of transformer_base hyper parameters set you can see in tensor2tensor's transformer.py module

ellurunaresh commented 6 years ago

I was training g2p model. My dictionary has 0.4M entries. I found as the number of steps increases my validation accuracy fluctuating. To overcome the overfitting problem, I tried to change the learning rate from 0.2 to 0.1 and also used dropout regularizer of 0.3 to hparams.relu_dropout, hparams.attention_dropout, hparams.layer_prepostprocess_dropout, hparams.symbol_dropout and hparams.dropout. The dropout gave worse than earlier results. Please provide me any solution to overcome this problem.

Does anybody faced this kind of problem?