how to calculate word embeddings like openai?

[hufuman]

Is there any way to create embeddings using LLMA2 as the base model?

[raghu-007]

import tensorflow as tf import llma2

model = tf.keras.models.load_model('llma2.h5')

embedding_layer = tf.keras.layers.Embedding( model.input_dim, model.output_dim, weights=[model.get_weights()[0]])

text_data = [['This is a sentence.'], ['This is another sentence.']] embeddings = embedding_layer(text_data)

embedding_vectors = embeddings.numpy()

You can try this code, I got to create embeddings from LLMA2 model from the Google AI Hub, and installed the tensorflow library.

[maziyarpanahi]

thanks @raghu-007 for sharing your approach. I wonder if this would be possible via HuggingFace's LlamaForCausalLM

[AlexandroLuis]

thanks @raghu-007 for sharing your approach. I wonder if this would be possible via HuggingFace's LlamaForCausalLM

maybe an approach like this?

from transformers import AutoModel, AutoTokenizer import torch

model_name = 'llma2.h5' tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModel.from_pretrained(model_name)

text_data = ['This is a sentence.', 'This is another sentence.']

encoded_input = tokenizer(text_data, padding=True, truncation=True, return_tensors='pt')

with torch.no_grad(): model_output = model(**encoded_input)

embeddings = model_output.last_hidden_state

embedding_vectors = embeddings.numpy()

[nparraa10]

How can I load the files or models in llma2? for example to llma-2-13b

[hjerbii]

The above code shared by @AlexandroLuis does not work for me :( . It returns a 2D array for every sentence. I wonder if there are some postprocessing steps to follow....