search

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
https://lmdeploy.readthedocs.io/en/latest/
Apache License 2.0
4.67k stars 428 forks source link

[Feature]- Support for the microsoft/Phi-3-vision-128k-instruct Vision Model #1637

Open sabarish244 opened 6 months ago

sabarish244 commented 6 months ago

Motivation

The latest release of microsoft phi3 4.2b 128k context vision model looks promising in performance and resource saving one too as it boast just 4.2b parameter. So it would be a great feature if the lmdeploy inference server supports it

Related resources

https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/tree/main https://azure.microsoft.com/en-us/blog/new-models-added-to-the-phi-3-family-available-on-microsoft-azure/

Additional context

I tried running the model via the lmdeploy docker inference server, installed the required additional packages and ran the model, model loaded and running but while trying to inference it via the api's we are getting either empty response or internal server error

RunningLeon commented 5 months ago

@sabarish244 Hi, thanks for your info. We include this model in the TODO list.

Youho99 commented 5 months ago

A first implementation on the vLLM project was made of Phi3-Vision.

Maybe this can help.

https://github.com/vllm-project/vllm/pull/4986

RunningLeon commented 4 months ago

@Youho99 hi, welcome to try this PR #1845 .