An analysis paper of BioBERT and BioELMo

dmis-lab / biobert

Bioinformatics'2020: BioBERT: a pre-trained biomedical language representation model for biomedical text mining

http://doi.org/10.1093/bioinformatics/btz682

Other

1.94k stars 451 forks source link

An analysis paper of BioBERT and BioELMo #28

Closed Andy-jqa closed 5 years ago

Andy-jqa commented 5 years ago

(Sorry to advertise our paper here)

Hi all, we have an analysis paper: Probing Biomedical Embeddings from Language Models, where we show how biomedical domain adaptation works for contextualized embeddings using probing tasks. Expectedly, fine-tuned BioBERT outperforms BioELMo in biomedical NER and NLI tasks. However, as fixed feature extractors BioELMo seems to be superior than BioBERT in our probing tasks.

In addition, we have BioELMo available at https://github.com/Andy-jqa/bioelmo.

jhyuklee commented 5 years ago

Nice work! Maybe you could specify which version of BioBERT you used. It seems that you used PubMed (200K), but PubMed + PMC (470K) worked best in our experiments. Thanks :)