Papers like https://arxiv.org/pdf/1910.10683.pdf show that further pre-training BERT on the target domain, on other languages etc. is generally useful under certain circumstances. In general, they state that pre-training + fine-tuning is always better than just fine-tuning and skipping pre-training.
This is something we could implement to improve our approach.
Papers like https://arxiv.org/pdf/1910.10683.pdf show that further pre-training BERT on the target domain, on other languages etc. is generally useful under certain circumstances. In general, they state that pre-training + fine-tuning is always better than just fine-tuning and skipping pre-training.
This is something we could implement to improve our approach.