neulab / ExplainaBoard

Interpretable Evaluation for AI Systems
MIT License
361 stars 36 forks source link

Test of analysis on open-domain QA #495

Open neubig opened 2 years ago

neubig commented 2 years ago

There is a nice PrimeQA framework developed by IBM, and it seems that it should make it relatively easy to generate open-domain QA results from multiple SOTA models.

It would be nice if we could look into these results, analyze them with explainaboard, and see if there are any ways we could use them to improve our analysis of Open Domain QA models.

In order to do this, we'd need to:

neubig commented 2 years ago

Suggestion from Avi at IBM, so I guess we should focus on PrimeQA's TyDiQA:

We’ve been focusing on TyDI for sometime now: the model’s available here too: https://huggingface.co/PrimeQA/tydiqa-primary-task-xlm-roberta-large. So if you want to do the same expts with TyDI and use PrimeQA you can totally get the system outputs by running decoding. It should be pretty straightforward.