search

agademic / custom-models-on-azure

Tutorial repository on how to deploy custom models on Azure.
3 stars 0 forks source link

How to deploy a custom HF model that not in the sentence-transformers list #1

Open vincent-revotech opened 5 months ago

vincent-revotech commented 5 months ago

Hi,

This notebook is awesome. I have a question, what if the model I want to deploy is not the sentence-transformers list, then how can I do that? For example, with this model https://huggingface.co/liuhaotian/llava-v1.5-7b Thanks,

agademic commented 5 months ago

Hi! Exactly the same way, you would use the model locally. Choose the appropriate framework, such as transformers adjust the score.py init to load the model (e.g. 

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "your/llm", device_map="auto", load_in_4bit=True
)

) and the code in score to make the predictions (e.g. model.generate().