Closed tomlimi closed 4 years ago
Hello,
Thanks for this project. I am looking for the method to get the embeddings from the selected encoder layer, instead of the last one. I've noticed that
out_layer_ndx
attribute should specify the indices of the layers which output is averaged. Is my attempt in the snippet below correct? If yes, maybe it could be included in the documentation. Otherwise, is there any way to do this right?import tensorflow as tf import bert n=6 bert_params = bert.params_from_pretrained_ckpt(modelBertDir) bert_layer = bert.BertModelLayer.from_params(bert_params, name="bert", out_layer_ndxs=[n]) model = tf.keras.Sequential([ tf.keras.layers.Input(shape=(128,), dtype='int32', name='input_ids'), self.bert_layer]) model.build(input_shape=(None, 128)) bert_layer.apply_adapter_freeze()
Hi! Thanks for the code
I'm just a Novice for BERT and TF codes And I will be grateful if you help my issue on your code
Do you mean that building bert_layer like that can make my code get embeddings from each hidden layers?
Than what does "n" in out_layer_ndxs[n] stands for? Is it the number of hidden layers from the model?
If so, I'm using 12 hidden layers and I understood about using out_layer_ndxs = [12]
But how can I finally see my embeddings by using out_layer_ndx ?????
Thank you for reading
out_layer_ndxs
is just a list of layer indexs, i.e. if it would be [1,2,11]
the output of the BertModelLayer
would be a tupple containing the outputs from the second, third and last layer.
out_layer_ndxs
is just a list of layer indexs, i.e. if it would be[1,2,11]
the output of theBertModelLayer
would be a tupple containing the outputs from the second, third and last layer.
Wow Thank you so much I finally can see my hidden layer's output
Hello,
Thanks for this project. I am looking for the method to get the embeddings from the selected encoder layer, instead of the last one. I've noticed that
out_layer_ndx
attribute should specify the indices of the layers which output is averaged. Is my attempt in the snippet below correct? If yes, maybe it could be included in the documentation. Otherwise, is there any way to do this right?