[11/15] 권세중, AlphaTuning (EMNLP 2022 Findings)

[sjquan]

When

• 11.15

  Who :

• 권세중 (sejung.kwon@navercorp.com)

  What

• Title: AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models
• Links: https://arxiv.org/abs/2210.03858

  Keywords:

• Large-scale PLM, Fine-tuning, Parameter-efficient adaptation, Quantization, NLG

  Preliminaries:

  • GPT-3
  • LoRA / (Hu, 2022) LoRA: Low-rank Adaptation of Large Language Models. ICLR 2022
  • P-tuning / (Liu, 2021) GPT Understands, Too. arXiv preprint arXiv:2103.10385.
  • nuqmm / (Park, 2022) nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models, arXiv:2206.09557

발표자료

[sjquan]

추가 ref. (Yao, 2022) ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers, Neurips 2022. (Dettmer, 2022) LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, Neurips 2022.