On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2020)

[kiccho1101]

https://arxiv.org/abs/2006.04884

Background

• Fine-tuning BERT, RoBERTa, and ALBERT are not fully known.
• Varying only the random seed leads to a large standard deviation of the fin-tuning accuracy.

Purpose

Main purpose is answering following question:

• Why is fine-tuning prone to failures and how can we improve its stability?

[kiccho1101]

[kiccho1101]