Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

[5g4s]

https://arxiv.org/abs/2002.11794

[5g4s]

[5g4s]

[5g4s]

We show that large models are more robust to compression techniques such as quantization and pruning than small models. Heavily compressed, large models achieve higher accuracy than lightly compressed, small models.

[5g4s]

[5g4s]

As models become increasingly large, they contain small subnetworks which achieve high accuracy.