Closed danielesposito27 closed 3 years ago
I believe, by calling build()
like this:
model.build((3, 128))
you are effectively telling the BertModelLayer
to expect a single tensor of token_ids
(with batch_size=3
) and use the default segment
type 0
:
If you want to specify segment
ids, i.e. token_type_ids
, try passing a list of shapes, i.e. something like:
l_input_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32')
l_token_type_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32')
# provide a custom token_type/segment id as a layer input
output = l_bert([l_input_ids, l_token_type_ids]) # [batch_size, max_seq_len, hidden_size]
model = keras.Model(inputs=[l_input_ids, l_token_type_ids], outputs=output)
model.build(input_shape=[(None, max_seq_len), (None, max_seq_len)])
Hey KPE!. Thank you for your answer, I implemented the code to accept the additional inputs. Is there a way to make it accept input_mask as a third input as well? Or should I include that in another way? it seems like the base code would only allow up to two. Thanks in advance and I apologize for asking again, any help is kindly appreciated (beginner here)!
@danielesposito27 - the mask is kind of auto generated internally (i.e. all zero token_ids get masked), and I believe, there is no way to explicitly specify it (but it should not be needed - in most cases I guess).
Hey everyone!. Thanks for this awesome repo
So I loaded some BERT weights that I obtained through customized BERT pretraining on a Spanish corpus. I loaded my weights as indicated by the team on previous issues:
So after I load my weights I try to pass it input data that looks like this:
As you know these are inputs from SQuAD dataset. Problem is that I get the following error msg;
By the way I don't get this error msg when I load a pretrained model from tensorflow hub but I need to use my own weights on this problem.
Thanks in advance!