search

zhiguowang / BiMPM

BiMPM: Bilateral Multi-Perspective Matching for Natural Language Sentences
Apache License 2.0
438 stars 150 forks source link

SentenceMatchDecoder.py gives a wrong answer #27

Open BluesBai opened 6 years ago

BluesBai commented 6 years ago

I trained my model and got a 82% accuracy on dev set. Then I tried to evaluated it on test set, but it gave me 37% accuracy. I thought it was very strange, so I also tried to run SentenceMatchDecoder.py on my dev set, it again gave a 36% accuracy, which is not reasonable because it should be 82%. Then I found the correct_tags in SentenceMatchTrainer.evaluate() actually claculated the number of pairs which originally have label '1' but not the correct # of predict. I tried to go through your code in SentenceMatchGraph.py but still have no idea of what's wrong. Can you help me to figure it out?

zhiguowang commented 6 years ago

I didn't meet this issue before. Please try out the new code. Let me know if you still have the issue.

zhengwsh commented 6 years ago

I have the same issue. The accuracy given by SentenceMatchDecoder.py is strange, compared with the accuracy trained. Have it been fixed?

netrookiecn commented 5 years ago

I also have the same issue, can u help us check this question

hexiahui commented 5 years ago

Hi, I meet the same issue. While trying, I got a 83% acc on dev set. However, when I run SentenceMatchDecoder.py. The dev set only got 65.6% acc.

hexiahui commented 5 years ago

I found the problem. Just set fix_word_vec = True. If not, u should save the embeddings during ur training.