[Project] Preparatory-Training (Placeholder name)

[zphang]

tl;dr: Add a second phase of pre-training to explicitly train models to be good at prompt-tuning and few-shot learning.

Full proposal: https://docs.google.com/document/d/1ER4tCfXO_Qn6GBWLi70sjrz-6kaMj8HCrTmFFnLf088/

Short version:

0. Choose a pretrained model e.g. GPT-J, T5, GLM
1. Build a large set of NLU and NLG tasks. Piggy-back off https://github.com/EleutherAI/lm_evaluation_harness/ where possible.
2. Do a second phase of pretraining (important: has to be pretraining scale, not fine-tuning scale) on massive task dataset, incorporating prompt tuning during training
3. Evaluate!

[StellaAthena]

@zphang To what extent would you say that this was scooped by FLAN and T0?