search

kongds / scaling_sentemb

Scaling Sentence Embeddings with Large Language Models
85 stars 4 forks source link

About the usage of your prompt. #14

Open only-changer opened 1 week ago

only-changer commented 1 week ago

Hi, I am interested in the prompt proposed in your paper to get sentence embeddings. I wonder if I can directly use your prompt in other open-source LLM, e.g. LLAMA 3, to get meaningful sentence embeddings?

For example, using this code to get the sentence embedding of "input_text", via the open-source LLAMA 3 without using the training part of your paper.

tokenizer = AutoTokenizer.from_pretrained(model_id) model = LlamaModel.from_pretrained(model_id,) prompt_template = 'This sentence: "{text}" means in one word:' text = prompt_template.format(text=input_text) t_input = tokenizer(text, return_tensors="pt").to(device) last_hidden_state = model(**t_input, output_hidden_states=True, return_dict=True).hidden_states[-1].cpu()

Thank you.

kongds commented 1 week ago

Yes, you can also test sentence embeddings with our prompt on LLaMA3 using the following scripts:

MODEL_PATH="meta-llama/Meta-Llama-3-8B"
TEMPLATE='This_sentence_:_"*sent_0*"_means_in_one_word:"'
python evaluation.py \
    --model_name_or_path   $MODEL_PATH \
    --mode test --mask_embedding_sentence \
    --mask_embedding_sentence_template $TEMPLATE --load_kbit 16

This will give the following results:

image