Open avivihadar opened 2 years ago
Thank you again for this great work @wetherbeei, could you please advise how to extract the activations of the intermediate layers of BERT?
@robert-srebrovic
Hi @robert-srebrovic, thanks for supporting this repo! my goal is to load the cls embeddings (including intermediate layers). So far I've attempted to use the bert-for-tf2 package. However, it seems like some of the layers defined here are missing from the target model when attempting to load it:
l_input_ids = keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype='int32')
bert_params = bert.params_from_pretrained_ckpt(MODEL_DIR)
l_bert = bert.BertModelLayer.from_params(bert_params, name="bert")
output = l_bert(l_input_ids)
model_chk = keras.Model(inputs=l_input_ids, outputs=output)
model_chk.build(input_shape=(None, MAX_SEQ_LENGTH))
bert.load_bert_weights(l_bert, model_ckpt)
Specifically, the following weights are missing:
bert/pooler/dense/bias
bert/pooler/dense/kernel
cls/predictions/output_bias
cls/predictions/transform/LayerNorm/beta
cls/predictions/transform/LayerNorm/gamma
cls/predictions/transform/dense/bias
cls/predictions/transform/dense/kernel
cls/seq_relationship/output_bias
cls/seq_relationship/output_weights
Could you please advise what would be the best way to reintroduce the missing layers?
Could you please provide an example code on how to access hidden layers?