Accessing representation of n-th layer

[tomlimi]

Hello,

Thanks for this project. I am looking for the method to get the embeddings from the selected encoder layer, instead of the last one. I've noticed that out_layer_ndx attribute should specify the indices of the layers which output is averaged. Is my attempt in the snippet below correct? If yes, maybe it could be included in the documentation. Otherwise, is there any way to do this right?

import tensorflow as tf
import bert

n=6
bert_params = bert.params_from_pretrained_ckpt(modelBertDir)
bert_layer = bert.BertModelLayer.from_params(bert_params, name="bert", out_layer_ndxs=[n])
model = tf.keras.Sequential([
            tf.keras.layers.Input(shape=(128,), dtype='int32', name='input_ids'),
            self.bert_layer])

model.build(input_shape=(None, 128))
bert_layer.apply_adapter_freeze()

[EJHyun]

Hello,

Thanks for this project. I am looking for the method to get the embeddings from the selected encoder layer, instead of the last one. I've noticed that out_layer_ndx attribute should specify the indices of the layers which output is averaged. Is my attempt in the snippet below correct? If yes, maybe it could be included in the documentation. Otherwise, is there any way to do this right?

import tensorflow as tf
import bert

n=6
bert_params = bert.params_from_pretrained_ckpt(modelBertDir)
bert_layer = bert.BertModelLayer.from_params(bert_params, name="bert", out_layer_ndxs=[n])
model = tf.keras.Sequential([
            tf.keras.layers.Input(shape=(128,), dtype='int32', name='input_ids'),
            self.bert_layer])

model.build(input_shape=(None, 128))
bert_layer.apply_adapter_freeze()

Hi! Thanks for the code

I'm just a Novice for BERT and TF codes And I will be grateful if you help my issue on your code

Do you mean that building bert_layer like that can make my code get embeddings from each hidden layers?

Than what does "n" in out_layer_ndxs[n] stands for? Is it the number of hidden layers from the model?

If so, I'm using 12 hidden layers and I understood about using out_layer_ndxs = [12]

But how can I finally see my embeddings by using out_layer_ndx ?????

Thank you for reading

[kpe]

out_layer_ndxs is just a list of layer indexs, i.e. if it would be [1,2,11] the output of the BertModelLayer would be a tupple containing the outputs from the second, third and last layer.

[EJHyun]

out_layer_ndxs is just a list of layer indexs, i.e. if it would be [1,2,11] the output of the BertModelLayer would be a tupple containing the outputs from the second, third and last layer.

Wow Thank you so much I finally can see my hidden layer's output