[Project] LM Eval Harness

[leogao2]

• Project: eval harness (https://github.com/EleutherAI/lm-evaluation-harness)
• Elevator Pitch: This project provides a unified framework to test autoregressive language models (GPT-2, GPT-3, GPTNeo, etc) on a large number of different evaluation tasks.
• Goal Outputs: An open source library and a paper about the library.
• Milestones: At a high level, what are the major milestones of this project (including completed ones)? Which ones are completed and which ones are in progress?
• How to Help: The main thing that can use help with is adding more task implementations. The task board is here: https://github.com/EleutherAI/lm-evaluation-harness/projects/1
• Desired Support: Mostly just need an unending stream of engineering effort to implement more tasks.

Other non-task things that need doing:

• [ ] Port most bigbench tasks over (see my half-finished impl in https://github.com/EleutherAI/lm-evaluation-harness/tree/bigbench)
• [ ] Optimizations
• [ ] Cleanup evaluator.py (it's currently a mess of spaghetti)