Using a Fine Tuned SciBERT

[navraj28]

I wanted to replace the Pre-trained SciBERT model with a Fine Tuned SciBERT Model. I achieved the Language Model Fine Tuning via this blog: https://github.com/Nikoschenk/language_model_finetuning/blob/master/scibert_fine_tuner.ipynb.

It uses the HuggingFace Library to achieve the Fine Tuning. The resulting HuggingFace Model has these components:

1. Config.json

2. Pytorch_model.bin

3. Special_tokens_map.json

4. Tokenizer_config.json

5. Vocab.txt

   I noticed that when I run Dygie’s get_scibert.py script, it downloads the Pytorch model as follows:

6. scibert_scivocab_cased/weights.tar.gz

7. scibert_scivocab_cased/vocab.txt

   Further, weights.tar.gz, is made up of pytorch_model.bin & bert_config.json.

   I repackaged HuggingFace outputs into the new weights.tar.gz (pytorch_model.bin & config.json renamed as bert_config.json).

   It works fine.

   I just wanted a second opinion, about the above approach. Pls advise.

[GillesJ]

I am not a maintainer, but for what it's worth I did the same with finBERT which is an in-domain BERT LM for my economic event extraction data and it seems to work fine too. For me, performance is not better than BERT-large though (F1 on TrigC and ArgC is off by 1-2%).

[dwadden]

Thanks GillesJ for the answer. I haven't actually tried this myself. If it seems to work, I'd say you're probably good. Closing unless other issues come up.