search

EmilyAlsentzer / clinicalBERT

repository for Publicly Available Clinical BERT Embeddings
MIT License
658 stars 134 forks source link

Pre Trained Model support for Tensorflow v2 #24

Closed dhanushanthp closed 4 years ago

dhanushanthp commented 4 years ago

I have converted the scripts from tf1 to tf2 and I'm trying to use one of your pre trained models for my pre-training. It's throwing Key bert/embeddings/layer_normalization/beta not found in checkpoint error. I understand that this error is cased by the one of the changed function in tensorflow v2. Where tf.contrib.layers.layer_norm(inputs=input_tensor, begin_norm_axis=-1, begin_params_axis=-1, scope=name) replaced by tf.keras.layers.LayerNormalization(axis=-1)(input_tensor). Btw this is in model.py line 364.

And without using the init_checkpoint everything works fine.

Therefore, I would like to check. Did you have any build model using tensorflow v2 with above LayerNormalizatoin Change.

EmilyAlsentzer commented 4 years ago

hi @dhanushanthp, can you provide a bit more detail here? Where is model.py?

Are you using the Huggingface repo to load clinicalBERT? If so, you might need to load the model as follows to use it with TF2.0:

model = TFAutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT", from_pt=True )

EmilyAlsentzer commented 4 years ago

Closing this due to inactivity. Feel free to reopen if your question has not been resolved.