3주차 논문 발표: DistilBERT, a distilled version of BERT_smaller, faster, cheaper and lighter

[najaehwa]

논문 제목

• DistilBERT, a distilled version of BERT : smaller, faster, cheaper and lighter.

논문 링크

• https://arxiv.org/pdf/1910.01108.pdf

발표 자료

• 링크

참조

• https://i-am-eden.tistory.com/46
• https://towardsdatascience.com/distillation-of-bert-like-models-the-theory-32e19a02641f
• https://cpm0722.github.io/paper-review/distilbert-a-distilled-version-of-bert-smaller-faster-cheaper-and-lighter
• https://aimaster.tistory.com/92
• https://seewoo5.tistory.com/18

[cwh1981]

teacher와 student layer 수가 같은 것 같은데, 왜 다른지라고 물어보신게 맞을까요?
https://github.com/monologg/DistilKoBERT/blob/master/distillation/train.py#L274

[najaehwa]

말씀하시는 vocab_size, hidden_size, max_position_embeddings 는 동일한 내용이 맞습니다. student와 teacher가 다른 부분은 Transformer Block layer 입니다.

혹시, 제가 설명을 잘못 드렸다면 아래와 같은 기준으로 참고하시면 좋을 것 같습니다.

base-cased 기준으로 살펴보면,,

• DistilKoBERT 기존의 12 layer를 3 layer로 줄였으며, 기타 configuration은 kobert를 그대로 따랐다고 하고, Layer 초기화의 경우 기존 KoBERT의 1, 5, 9번째 layer 값을 그대로 사용하였다고 합니다. https://github.com/monologg/DistilKoBERT#pretraining-distilkobert

• DistilKoBERT config ("n_layers": 3) https://github.com/monologg/DistilKoBERT/blob/5a2b750cbe1e9ca3fe65d91d42f7e0034bf1c08a/distillation/training_configs/distilkobert_3_layer.json#L10

• DistilBERT The model has 6 layers, 768 dimension and 12 heads, totalizing 65M parameters. https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation#how-to-use-distilbert

• DistilBERT config ("n_layers": 6) https://github.com/huggingface/transformers/blob/3b00b623b7cad9e1b7c71c97fff24a0286b37045/examples/research_projects/distillation/training_configs/distilbert-base-cased.json#L10

• bert-base-cased config ("num_hidden_layers": 12) https://huggingface.co/bert-base-cased/blob/main/config.json#L16