search

tarrade / proj_multilingual_text_classification

Explore multilingal text classification using embedding, bert and deep learning architecture
Apache License 2.0
4 stars 1 forks source link

Further Pre-training: do-able? Useful? #58

Closed vluechinger closed 4 years ago

vluechinger commented 4 years ago

Papers like https://arxiv.org/pdf/1910.10683.pdf show that further pre-training BERT on the target domain, on other languages etc. is generally useful under certain circumstances. In general, they state that pre-training + fine-tuning is always better than just fine-tuning and skipping pre-training.

This is something we could implement to improve our approach.

tarrade commented 4 years ago

Will take too much time and cost too much for a little impact. To see with other project when we need AXA data and insurance context