How to access hidden layers?

[avivihadar]

Could you please provide an example code on how to access hidden layers?

[avivihadar]

Thank you again for this great work @wetherbeei, could you please advise how to extract the activations of the intermediate layers of BERT?

[wetherbeei]

@robert-srebrovic

[avivihadar]

Hi @robert-srebrovic, thanks for supporting this repo! my goal is to load the cls embeddings (including intermediate layers). So far I've attempted to use the bert-for-tf2 package. However, it seems like some of the layers defined here are missing from the target model when attempting to load it:

l_input_ids  = keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype='int32')

bert_params = bert.params_from_pretrained_ckpt(MODEL_DIR)
l_bert = bert.BertModelLayer.from_params(bert_params, name="bert")

output = l_bert(l_input_ids)
model_chk = keras.Model(inputs=l_input_ids, outputs=output)
model_chk.build(input_shape=(None, MAX_SEQ_LENGTH))
bert.load_bert_weights(l_bert, model_ckpt)

Specifically, the following weights are missing:

bert/pooler/dense/bias
bert/pooler/dense/kernel
cls/predictions/output_bias
cls/predictions/transform/LayerNorm/beta
cls/predictions/transform/LayerNorm/gamma
cls/predictions/transform/dense/bias
cls/predictions/transform/dense/kernel
cls/seq_relationship/output_bias
cls/seq_relationship/output_weights

Could you please advise what would be the best way to reintroduce the missing layers?