search

easonnie / semanticRetrievalMRS

This is the repo for the paper "Revealing the Importance of Semantic Retrieval for Machine Reading at Scale".
MIT License
59 stars 11 forks source link

Low accuracy #6

Open dujiaxin opened 4 years ago

dujiaxin commented 4 years ago

This looks like a wonderful model. However, when I run your model according to your README.md, it gave me em,f1,prec,recall all about 0.04. Is there any thing I should change besides default settings?

anushkmittal commented 4 years ago

@dujiaxin were you able to figure it out?

cdmalon commented 4 years ago

Same problem:

{'label': 'ema', 'score': {'em': 0.041188386225523295, 'f1': 0.04091829844699527, 'prec': 0.04091829844699527, 'recall': 0.04091829844699527, 'doc_em': 0.0, 'doc_f1': 0.0, 'doc_prec': 0.0, 'doc_recall': 0.0, 'sp_em': 0.0, 'sp_f1': 0.0, 'sp_prec': 0.0, 'sp_recall': 0.0, 'joint_em': 0.0, 'joint_f1': 0.0, 'joint_prec': 0.0, 'joint_recall': 0.0}}

nixiaoyu commented 3 years ago

I met the same problem. Have you solved it?

{'label': 'ema', 'score': {'em': 0.04010803511141121, 'f1': 0.04010803511141121, 'prec': 0.04010803511141121, 'recall': 0.04010803511141121, 'doc_em': 0.0, 'doc_f1': 0.0, 'doc_prec': 0.0, 'doc_recall': 0.0, 'sp_em': 0.0, 'sp_f1': 0.0, 'sp_prec': 0.0, 'sp_recall': 0.0, 'joint_em': 0.0, 'joint_f1': 0.0, 'joint_prec': 0.0, 'joint_recall': 0.0}}

easonnie commented 3 years ago

You can try to change the seed in the following code snippet: https://github.com/easonnie/semanticRetrievalMRS/blob/149833a7ca9146ef4a89d90e0b6cedd0ab088e72/src/qa_models/hotpot_bert_v0.py#L130