If I want to use another LLM model, which parts of the code do I need to customize?

[liuweie]

hi，I am trying to use llm2vec with another LLM model ( not Mistral or LLama),which parts of the code should I customize? I personally think that the code in the models section of the llm2vec library should be modified, such as adding a code similar to bidirectional_llama.py, am I right?

thx

[vaibhavad]

Yes, that is correct, some portions of loading the model will also need to be changed https://github.com/McGill-NLP/llm2vec/blob/main/llm2vec/llm2vec.py#L59

Finally, if you want to define a custom template, it will require changes here. In the future versions of the library, the template will be specified outside the package (#56 )

Let me know if you have any further questions.

[liuweie]

Yes, that is correct, some portions of loading the model will also need to be changed https://github.com/McGill-NLP/llm2vec/blob/main/llm2vec/llm2vec.py#L59

Finally, if you want to define a custom template, it will require changes here. In the future versions of the library, the template will be specified outside the package (#56 )

Let me know if you have any further questions.

thanks for your reply! By the way, I found that if I want to use my customized dadaset, I also need to add a dataset code in llm2vec package, just like E5data.py in dataset module. I think it is not a flexible way , Is there another way to do this ?

[ciekawy]

I tried the phi3 and while I managed to convert, I see the tensor names changed, e.g. the "model." prefix is missing etc...

[vaibhavad]

Do you have a fork where I can see the code?

[vaibhavad]

Closing as it is stale. Feel free to re-open if you have any more questions.