search

microsoft / KEAR

Official code for achieving human parity on CommonsenseQA with External Attention
106 stars 25 forks source link

Missing preprocessing script? #9

Closed zixinzeng-jennifer closed 2 years ago

zixinzeng-jennifer commented 2 years ago

Hi, I'm trying to run your model on a similar QA dataset, but I am wondering which script you used for generating the concepts in question & choices.

xycforgithub commented 2 years ago

Hi @zixinzeng-jennifer, The concepts in CommonsenseQA are provided as a part of the dataset, so we do not generate the concepts for question or choices.

zixinzeng-jennifer commented 2 years ago

I see, thanks for the reply! The Entity Linker in spacy can be used for preprocessing concepts (according to wikidata).