search

dmis-lab / biobert

Bioinformatics'2020: BioBERT: a pre-trained biomedical language representation model for biomedical text mining
http://doi.org/10.1093/bioinformatics/btz682
Other
1.93k stars 451 forks source link

biobert for keras version of huggingface transformers #98

Closed dmollaaliod closed 4 years ago

dmollaaliod commented 4 years ago

Hi, are there any instructions/hints about how to load biobert as a keras layer using the huggingface transformers? I tried several possibilities but none of these worked. All that I found out is how to use the pytorch version but I am interested in the keras layer version. Below are two of my attempts (I saved the biobert files into folder "biobert_v1.1_pubmed").

Attempt 1:

biobert_model = TFBertModel.from_pretrained('bert-base-uncased')
biobert_model.load_weights('biobert_v1.1_pubmed/model.ckpt-1000000')

Error message:

AssertionError: Some objects had attributes which were not restored:
    : ['tf_bert_model_4/bert/embeddings/word_embeddings/weight']
    : ['tf_bert_model_4/bert/embeddings/position_embeddings/embeddings']
   (and many more lines like above...)

Attempt 2:

biobert_model = TFBertModel.from_pretrained("biobert_v1.1_pubmed/model.ckpt-1000000", config='biobert_v1.1_pubmed/bert_config.json')

Error message:

NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights).

Any help appreciated! My experience with huggingface's transformers library is almost zero.

dmollaaliod commented 4 years ago

Someone answered in stackoverflow, it worked for me :)