gbert - advanced training, adaptive fine-tuning

dominikmn / one-million-posts

Assisting newspaper moderators with machine learning.

MIT License

2 stars 1 forks source link

gbert - advanced training, adaptive fine-tuning #131

Open dominikmn opened 3 years ago

dominikmn commented 3 years ago

Train gbert with a masked language model task (MLM task) making use of the 1 million unlabeled posts of the OMP data set.

Ressources

Section "Adaptive fine-tuning" https://ruder.io/recent-advances-lm-fine-tuning/
BERT paper: https://arxiv.org/abs/1810.04805

dominikmn commented 3 years ago

The first two commits 81c452dbbb818a6b3bb2356eaea0b5cc55b4d913 and 4a2a39aad37b3b9eb64ccbbfac21125ffbd0cb01 combined are an alternative solution to #125. They are not yet related to the actual adaptive fine-tuning.