Open LeenaShekhar opened 7 years ago
I am facing with the exact same problem, can someone guide us.
I have not worked on it after that, but from experience it might be because of the training. Usually you need t train for longer steps in such models. Try that and see it that works. I have not explored the pre-trained models tough. You can compare your output with that.
I trained the network for 5000 iterations; see the loss below
Model saved to disk at iteration #1000 val loss : 3.290424
Model saved to disk at iteration #2000 val loss : 3.261373
Model saved to disk at iteration #3000 val loss : 3.224990
Model saved to disk at iteration #4000 val loss : 3.151570
Model saved to disk at iteration #5000 val loss : 3.155647
After this I wanted to evaluate the model on the test dataset. Most of the decoder's output is "unk" (see below):
q : [hillary is crazy also evil nothing good about her except that she has a terminal illness]; a : [i unk unk unk unk unk unk unk unk unk unk unk unk unk] q : [breaking unk unk israeli unk and unk peace prize winner dies at unk]; a : [unk unk unk unk unk unk unk unk unk unk unk unk unk] q : [because and jason unk are fighting in the cage next week to see who unk into whom]; a : [i unk unk unk unk unk unk unk unk unk unk unk unk unk] q : [im considering unk a ticket shit looks live ]; a : [i unk unk unk] q : [unk is a classic but tears in heaven is stupid]; a : [i unk unk unk unk unk unk unk unk unk unk unk unk unk]
Do you think the output is like that because I tested the model's performance too soon or that the model is not learning anything?