search

McGill-NLP / llm2vec

Code for 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders'
https://mcgill-nlp.github.io/llm2vec/
MIT License
1.31k stars 95 forks source link

Building with `pretrained_model` not `instruct_model`? #120

Closed su-park closed 4 months ago

su-park commented 4 months ago

Hello. I have one more question.

When building models, You primarily have used instruct models, but is it also possible to use pretrained models? Is there a performance difference when building with pretrained_model and instruct_model of the same model or any special reasons to prefer instruct_model?

vaibhavad commented 4 months ago

Hi @su-park,

we used instruct models because the evaluation is done using instructions, hence, instruction models are better suited for the task than the pretrained models of the same model family. We have not yet experimented with using pretrained_model and instruct_model of the same model.