As BERT and language models in general are rather huge, it is worth thinking about smaller versions, especially when it comes to deployment. This heavily depends on the later use case where the models are applied. Whenever possible, larger models should give a better result whilst using up more resources.
The basic functionality of distilled models follows a teacher-student architecture where attention heads are removed (detailed knowledge should be acquired).
As BERT and language models in general are rather huge, it is worth thinking about smaller versions, especially when it comes to deployment. This heavily depends on the later use case where the models are applied. Whenever possible, larger models should give a better result whilst using up more resources.
The basic functionality of distilled models follows a teacher-student architecture where attention heads are removed (detailed knowledge should be acquired).
Open questions: