BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

[arxiv-survey-bot[bot]]

URL: http://arxiv.org/abs/2002.02925v2

In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models, and smooths the training process. Compared to the previous knowledge distillation approaches for BERT compression, our approach leverages only one loss function and one hyper-parameter, liberating human effort from hyper-parameter tuning. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.

[iron316]

1. 問題設定と解決した点

BERTはたくさんの良い結果を示すがモデルが大きすぎるため，BERTを蒸留する研究は加発に行われてきた， 本論文ではオリジナルのBERTの精度を維持しつつモデルを圧縮する手法を提案する．

2. 技術や手法のキモ

本手法ではBERTのn, n+1層を1層に圧縮するBERT-of-Theseusを使用する，

3. 主張の有効性検証

4. 議論・所感

5. 次に読むべき論文

6. 参考文献

7. 補足