search

haichao592 / UnAnsQ

Resources of ACL2019 paper "Learning to Ask Unanswerable Questions for Machine Reading Comprehension"
4 stars 0 forks source link

QA Data #2

Closed WWWWWLI closed 5 years ago

WWWWWLI commented 5 years ago

Hi, I use ''pair-AB-passage-augmentation-v2.0-uncased-best1.json'' file to train the bert, but result don't improve.

haichao592 commented 5 years ago

Please provide all the essential details if you need help.

WWWWWLI commented 5 years ago

Sorry, I will tell you my experiment details. I use pre-train model bert, bert-base-uncased. As contrast, I continue train two epoch on SQuAD trainv-2.0, as baseline. The dev result is "exact": 73.09020466604902, "f1": 76.25957836849379, "total": 11873, "HasAns_exact": 72.11538461538461, "HasAns_f1": 78.4632209799471, "HasAns_total": 5928, "NoAns_exact": 74.06223717409588, "NoAns_f1": 74.06223717409588, "NoAns_total": 5945,

And I use your augmentation data(pair-AB-passage-augmentation-v2.0-uncased-best1.json) to train bert-base-uncased, two epoch(the same as baseline). The dev result is "exact": 73.09020466604902, "f1": 76.25957836849379, "total": 11873, "HasAns_exact": 72.11538461538461, "HasAns_f1": 78.4632209799471, "HasAns_total": 5928, "NoAns_exact": 74.06223717409588, "NoAns_f1": 74.06223717409588, "NoAns_total": 5945,

The result is same.

haichao592 commented 5 years ago

Could you tell me how you use the augmentation data? You should check your code since the results can not be exactly same and your baseline results are too low.

WWWWWLI commented 5 years ago

I just direct use the augmentation data, the command like this python ./examples/run_squad.py ----train_file ./datasets/SQuAD/pair-AB-passage-augmentation-v2.0-uncased-best1.json . By the way, I not add the data in train-2.0.json to pair-AB-passage-augmentation-v2.0-uncased-best1.json .

I think result of baseline is not important, I just want contrast the data augmentation influence.

haichao592 commented 5 years ago

Firstly, you did not use the augmentation data correctly. We have the correct way wrote in the paper please read the paper more carefully.

Secondly, the results of the baseline are important. It implies you have get the prerequistes done correctly and it is the base of further experiments, which is important.

Thank you.