Open Benjamim-EP opened 3 years ago
Hi @Benjamim-EP ,
I am not a TensorFlow user, so unfortunately I can't give you directions. But it should be possible to adapt a working example for English BERT (or other language) using BERTimbau TensorFlow checkpoint (weights) and config file. I'll leave this issue open so others may help you. Please share your experience with us if you find a solution :)
Hi @Benjamim-EP and @fabiocapsouza , using this model as part of another model should be straightforward, here is a minimal example, that uses BERT as base and adds a classifier head at the top:
# Load BERT with the HF API
encoder = TFBertModel.from_pretrained('path/to/bert_dir/', from_pt=True)
# Build model composed with BERT
# Input layers (from the tokenizer)
input_ids = tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name='input_ids')
token_type_ids = tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name='token_type_ids')
attention_mask = tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name='attention_mask')
# BERT encoder
encoded = encoder({"input_ids": input_ids,
"token_type_ids": token_type_ids,
"attention_mask": attention_mask})['pooler_output']
# Classifier head
outputs = tf.keras.layers.Dense(n_classes, activation='softmax', name='classifier')(encoded)
# Build the model
model = tf.keras.models.Model(inputs=[input_ids, token_type_ids, attention_mask], outputs=outputs)
The only issue I faced is that the TFBertModel
is not able to load the files from the Tensorflow
checkpoints, so you need to load from PyTorch
and use from_pt=True
I believe this example should be in the README as an example of how to use it with Tensorflow. ;)
I would like to know how I can use this template as in the example below
` class DCNNBERTEmbedding(tf.keras.Model):
`