search

kevinduh / san_mrc

Stochastic Answer Networks (SAN) for Machine Reading Comprehension
BSD 3-Clause "New" or "Revised" License
148 stars 47 forks source link

About classifier #18

Closed softexplorer closed 5 years ago

softexplorer commented 5 years ago

Where do you use the classifier in the code of predicting? How the classifer affect output "devoutput**.json"?

namisan commented 5 years ago

I didn't understand your questions. Could you explain it more details?

softexplorer commented 5 years ago

The prediction can be got by the function predict_squad(model, dev_data, v2_on=args.v2_on). I can't find where the result of classifier affect prediction.In function model.predict, if s_idx == len(spans[i]) - 1: answer = '', but the result of classifier 'lab' don't decide if answer = ‘’.Where do the result of classifier decide if answer = ''.

namisan commented 5 years ago

If the index of answer points the last dumpy token, it means that it is an empty answer as attached line. https://github.com/kevinduh/san_mrc/blob/master/src/model.py#L141

To utilize results of classifier, you may need to select a threshold based on dev by calling the official evaluation script. As the attached example, if we select the threshold 0.2824*, it gives best F1 70.1. Hope this answers your questions.

OrderedDict([('exact', 66.29327044554873), ('f1', 69.08254766223166), ('total', 11873), ('HasAns_exact', 61.94331983805668), ('HasAns_f1', 67.52987321080874), ('HasAns_total', 5928), ('NoAns_exact', 70.63078216989067), ('NoAns_f1', 70.63078216989067), ('NoAns_total', 5945), ('best_exact', 67.48926134927987), ('best_exact_thresh', 0.26524031162261963), ('best_f1', 70.0682365711497), ('best_f1_thresh', 0.2824656665325165)])