[11/29] 김민수, Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders

[MarsJacobs]

Date

• 2022.11.29

Who

• 김민수 (minsoo2333@hanyang.ac.kr)

Keywords

• Quantization-Aware Training (QAT), Transformer, Knowledge Distillation

What

• Understanding and Improving KD for QAT of Large Transformer Encoders (EMNLP 2022)
  • https://arxiv.org/abs/2211.11014
  • Analyze and propose new sets of KD functions for QAT with Transformer Encoders

Preliminary

Transformer 계열 모델 (주로 Encoder) 에 QAT 가 적용된 최신 연구들을 차례차례 살펴볼 예정입니다.

• TernaryBERT (EMNLP 2020)
  • https://arxiv.org/abs/2009.12812
  • KD approach with QAT to BERT
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization (EMNLP 2021)
  • https://arxiv.org/abs/2109.12948
  • Analyze the challenges of Quantization with Transformer
  • XTC (NeurIPS 2022)
  • https://arxiv.org/abs/2206.01859
  • Exploration of QAT settings to extremely low-bit precision BERT
• DQ-BART (ACL 2022)
  • https://arxiv.org/abs/2203.11239
  • Employing TernaryBERT methodology to Seq2Seq model (BART) with model compression
• QuantGPT (ACL 2022 - Outstanding Papers)
  • https://arxiv.org/abs/2203.10705
  • KD (Contrastive) approach with GPT-2