search

lucko515 / speech-recognition-neural-network

This is the end-to-end Speech Recognition neural network, deployed in Keras. This was my final project for Artificial Intelligence Nanodegree @Udacity.
187 stars 87 forks source link

Validation loss fluctuating #3

Open GauriDhande opened 6 years ago

GauriDhande commented 6 years ago

Iam trying to train the model_end with few hyperparameter tuning changes. Iam also training on my own dataset of 600 wavefiles, splited as 10% test dataset, 10% validation dataset and 80% training dataset.

def train_model(input_to_softmax, pickle_path, save_model_path, train_json='train_corpus.json', valid_json='valid_corpus.json', test_json='test_corpus.json', minibatch_size=8, spectrogram=True, mfcc_dim=13, optimizer=Adam(lr=0.0001, decay=1e-6), epochs=3000, verbose=1, sort_by_duration=False, max_duration=18.0):

model summary:

Layer (type) Output Shape Param #

the_input (InputLayer) (None, None, 13) 0

layer_1_conv (Conv1D) (None, None, 100) 39100

conv_batch_norm (BatchNormal (None, None, 100) 400

rnn_1 (GRU) (None, None, 100) 60300

bt_rnn_1 (BatchNormalization (None, None, 100) 400

rnn_bi (GRU) (None, None, 100) 60300

bt_rnn_bi (BatchNormalizatio (None, None, 100) 400

time_distributed_6 (TimeDist (None, None, 29) 2929

softmax (Activation) (None, None, 29) 0

Total params: 163,829 Trainable params: 163,229 Non-trainable params: 600

None

First 20 epochs: Epoch 1/3000 69/70 [============================>.] - ETA: 0s - loss: 828.7257 - acc: 0.0000e+00Epoch 00000: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 824.4764 - acc: 0.0000e+00 - val_loss: 1025.0829 - val_acc: 0.0000e+00 Epoch 2/3000 69/70 [============================>.] - ETA: 0s - loss: 576.0773 - acc: 0.0000e+00Epoch 00001: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 574.0380 - acc: 0.0000e+00 - val_loss: 965.9543 - val_acc: 0.0000e+00 Epoch 3/3000 69/70 [============================>.] - ETA: 0s - loss: 475.5182 - acc: 0.0000e+00Epoch 00002: saving model to results/model_end.h5 70/70 [==============================] - 71s - loss: 475.8133 - acc: 0.0000e+00 - val_loss: 840.2710 - val_acc: 0.0000e+00 Epoch 4/3000 69/70 [============================>.] - ETA: 0s - loss: 446.0698 - acc: 0.0000e+00Epoch 00003: saving model to results/model_end.h5 70/70 [==============================] - 70s - loss: 446.3768 - acc: 0.0000e+00 - val_loss: 649.6033 - val_acc: 0.0000e+00 Epoch 5/3000 69/70 [============================>.] - ETA: 0s - loss: 420.5421 - acc: 0.0000e+00Epoch 00004: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 420.1927 - acc: 0.0000e+00 - val_loss: 505.9156 - val_acc: 0.0000e+00 Epoch 6/3000 69/70 [============================>.] - ETA: 0s - loss: 412.7203 - acc: 0.0000e+00Epoch 00005: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 412.8178 - acc: 0.0000e+00 - val_loss: 450.0329 - val_acc: 0.0000e+00 Epoch 7/3000 69/70 [============================>.] - ETA: 0s - loss: 397.5730 - acc: 0.0000e+00Epoch 00006: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 398.1866 - acc: 0.0000e+00 - val_loss: 395.4442 - val_acc: 0.0000e+00 Epoch 8/3000 69/70 [============================>.] - ETA: 0s - loss: 388.5409 - acc: 0.0000e+00Epoch 00007: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 386.8799 - acc: 0.0000e+00 - val_loss: 401.9153 - val_acc: 0.0000e+00 Epoch 9/3000 69/70 [============================>.] - ETA: 0s - loss: 389.5977 - acc: 0.0000e+00Epoch 00008: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 389.4545 - acc: 0.0000e+00 - val_loss: 388.4293 - val_acc: 0.0000e+00 Epoch 10/3000 69/70 [============================>.] - ETA: 0s - loss: 378.1360 - acc: 0.0000e+00Epoch 00009: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 376.7665 - acc: 0.0000e+00 - val_loss: 407.0841 - val_acc: 0.0000e+00 Epoch 11/3000 69/70 [============================>.] - ETA: 0s - loss: 374.1938 - acc: 0.0000e+00Epoch 00010: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 374.0701 - acc: 0.0000e+00 - val_loss: 361.8077 - val_acc: 0.0000e+00 Epoch 12/3000 69/70 [============================>.] - ETA: 0s - loss: 373.0912 - acc: 0.0000e+00Epoch 00011: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 373.9879 - acc: 0.0000e+00 - val_loss: 362.8776 - val_acc: 0.0000e+00 Epoch 13/3000 69/70 [============================>.] - ETA: 0s - loss: 370.4228 - acc: 0.0000e+00Epoch 00012: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 370.9717 - acc: 0.0000e+00 - val_loss: 353.5565 - val_acc: 0.0000e+00 Epoch 14/3000 69/70 [============================>.] - ETA: 0s - loss: 363.2626 - acc: 0.0000e+00Epoch 00013: saving model to results/model_end.h5 70/70 [==============================] - 69s - loss: 364.6332 - acc: 0.0000e+00 - val_loss: 350.5256 - val_acc: 0.0000e+00 Epoch 15/3000 69/70 [============================>.] - ETA: 0s - loss: 361.7289 - acc: 0.0000e+00Epoch 00014: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 362.7544 - acc: 0.0000e+00 - val_loss: 391.8794 - val_acc: 0.0000e+00 Epoch 16/3000 69/70 [============================>.] - ETA: 0s - loss: 360.1477 - acc: 0.0000e+00Epoch 00015: saving model to results/model_end.h5 70/70 [==============================] - 70s - loss: 358.5634 - acc: 0.0000e+00 - val_loss: 389.8897 - val_acc: 0.0000e+00 Epoch 17/3000 69/70 [============================>.] - ETA: 0s - loss: 363.4254 - acc: 0.0000e+00Epoch 00016: saving model to results/model_end.h5 70/70 [==============================] - 70s - loss: 362.3484 - acc: 0.0000e+00 - val_loss: 347.7054 - val_acc: 0.0000e+00 Epoch 18/3000 69/70 [============================>.] - ETA: 0s - loss: 358.4653 - acc: 0.0000e+00Epoch 00017: saving model to results/model_end.h5 70/70 [==============================] - 70s - loss: 357.7992 - acc: 0.0000e+00 - val_loss: 382.3785 - val_acc: 0.0000e+00 Epoch 19/3000 69/70 [============================>.] - ETA: 0s - loss: 355.4213 - acc: 0.0000e+00Epoch 00018: saving model to results/model_end.h5 70/70 [==============================] - 68s - loss: 355.0019 - acc: 0.0000e+00 - val_loss: 378.6394 - val_acc: 0.0000e+00 Epoch 20/3000 69/70 [============================>.] - ETA: 0s - loss: 356.1490 - acc: 0.0000e+00Epoch 00019: saving model to results/model_end.h5 70/70 [==============================] - 70s - loss: 355.6650 - acc: 0.0000e+00 - val_loss: 369.9678 - val_acc: 0.0000e+00

as you can see, the training and validation loss is very high an also vlidation loss is going up and down. Also, can anyone suggest how this loss can be reduced?

Savant-HO commented 1 year ago

Hello, I also meet this problem. Could you please tell me how to solve it? Thanks!