search

AlexGidiotis / Multimodal-Gesture-Recognition-with-LSTMs-and-CTC

An end-to-end system that performs temporal recognition of gesture sequences using speech and skeletal input. The model combines three networks with a CTC output layer that recognises gestures from continuous stream.
MIT License
28 stars 5 forks source link

About the results of model prediction #2

Open likestudy opened 5 years ago

likestudy commented 5 years ago

Hello everyone: I am trying to do this experiment as the REAME said, but during training the audio network, the loss of this model is around 150¡£As I tired to use this model to predict something, I can get nothing but 'sil'. Is there any problems with the features that I extracted, or I used this model in a wrong way? But the features that I use are just the 39 MFCCs that extracted by the HCopy.

Has anyone ever meet this problem? Or can anyone tell me the result of your models? I'm so confused, but I can't figure out the problem.

Here I attach a file's features that I extracted. Thanks very much . audio_1.csv.zip

AlexGidiotis commented 5 years ago

I think that since you are getting output you are using the model correctly. The issue here is that the training hasn't converged. CTC is usually tricky to optimise so the training might not be as smooth as with other loss functions. Unfortunately, I cannot tell if the MFCC features are correct but if you followed the instructions they are most likely fine.

With that said I think you should try the following:

  1. Make sure that your labels are correctly assigned during the training.

  2. Once the training loss stops reducing for a few epochs (10-20) try interrupting the training and restarting from the checkpoint.

  3. Play around with the learning rate and the clipping value.

  4. Increase the early stopping patience.