Open babla9 opened 1 month ago
@babla9 vLLM and mistral.rs both supports phi3-vision. But, merging the lora weights should be more easier, when you use it.
@2U1 What do you mean by merging the lora weights? Lora for phi-3 vision is currently not supported in vLLM.
@kevintee When you fine tune the model with the code. You will get a adapter weight(Because it finetunes phi3 with lora). You should merge in to the original model to use it with vLLM.
Thanks so much for your work on this!
How can I deploy this fine-tuned model (expose via API endpoint)? Can I use vLLM or a library like this: https://github.com/EricLBuehler/mistral.rs, which supports Phi3-Vision?
Thanks!