search

allenai / allennlp-reading-comprehension

81 stars 25 forks source link

Add BERT QA #13

Closed pengshuang closed 4 years ago

pengshuang commented 4 years ago

see issue https://github.com/allenai/allennlp-reading-comprehension/issues/10

@dirkgr This is my implementation of BERT-QA (the UT is coming soon), but I have some problems when compared with the official huggingface's version. The F1 and EM I got are lower in SQuAD 1.1 dataset.

F1-Score Comparison Huggingface: 88.0% Ours: 82.1%

Interestingly, the comparison performance in Chinese MRC tasks are consistent. I have a hard time to figure the problem (but failed) and thus hope to get your suggestions.

Feel free to contact me if you have any ideas.

dirkgr commented 4 years ago

This code also doesn't add a CLS token, but I guess that's not strictly necessary for this problem.

pengshuang commented 4 years ago

This code also doesn't add a CLS token, but I guess that's not strictly necessary for this problem.

Actually, here we don't need to add a CLS token manually, the token_indexers will automatically add the CLS and SEP tokens later.

pengshuang commented 4 years ago

I have found the reason that accounts for the difference in score and will fix it with UT in this weekend.

dirkgr commented 4 years ago

I'm closing this for lack of activity, and also because I already took the good bits and put them into #19. @pengshuang, even though we're not merging this, thanks for the contribution! I stole a few things from here.