FINETUNED LANGUAGE MODELS ARE ZERO-SHOT LEARNERS

[jinmang2]

집현전 최신반 스터디

• 2022년 4월 24일 일요일 10시
• 최규담님 오지은님 최민주님 발표
• 논문 링크: https://arxiv.org/abs/2109.01652

Abstract

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

[jinmang2]

• 발표 링크: https://www.youtube.com/watch?v=HvIZVm0j_E4&list=PL2tRglRS_GqivYoDuzLNo_HezOvh4Z1XN&index=6