Add TyDi QA to Simple Benchmark - Githubissues

bigscience-workshop / evaluation

Code and Data for Evaluation WG

Other

41 stars 24 forks source link

Add TyDi QA to Simple Benchmark #44

Closed wilsonyhlee closed 3 years ago

wilsonyhlee commented 3 years ago

This PR adds the TyDi QA dataset to the simple benchmark script, including

Load TyDi QA and reformat the validation dataset for the secondary task into the prompt provided by the prompt WG
Generate predictions with a decoder model (ie GPT-2)
Compute and output substring_match metric

Estimated runtime on GPU is ~1 minute; CPU ~15 minutes.