Add ANLI dataset - Githubissues

bigscience-workshop / evaluation

Code and Data for Evaluation WG

Other

41 stars 24 forks source link

Add ANLI dataset #71

Open omerant opened 2 years ago

omerant commented 2 years ago

My attempt to add the ANLI dataset (issue #32), including:

Load ANLI and reformat each of the three validation splits (R1, R2, R3) into the prompt provided by the prompt WG
Generate predictions with an autoregressive model (i.e. GPT-2)
Compute and output accuracy per split

Estimated runtime on GPU is ~30 seconds; evaluated on GPT2.