Support Multiple Models

[aldrinc]

• Allow user to specify multiple models to download when loading server
• Allow user to switch between models
• Allow user to load multiple models on the cluster (nice to have)

[zhuohan123]

For first and second feature request, why don't you just kill the old server and start a new one with the new model?

[aldrinc]

It’s what we are doing now but takes long time to download and load large models (33/65b). For train to deploy pipeline not possible to have zero downtime without multiple servers and blue green deployment strategy.

[zhuohan123]

The downloading should only happen at the very first time when using a model. However, the loading cost is unavoidable. Are you looking for something that can swap the model with zero downtime?

[gesanqiu]

I think multi-models is important to some bussiness logic like ensemble model and langchain application, do you have any ideas I can reference, then I will try to implement it on vLLM.

[ft-algo]

Is there any progress on this feature?

[shixianc]

+1

[Ki6an]

+1

[corticalstack]

+1 For enterprise, instead of one monolithic, API-based LLM like GPT4, their strategy may be a collection of SLMs dedicated/fine-tuned to specific tasks. THis is why they will want to serve multiple models simultaneously, for e.g. phi2/mistral-7B/yi-34B. Could we have an update on this feature request please?

[capybarahero]

+1

[chanchimin]

+1 how is the progress of this feature

[lenartgolob]

+1

[Shamepoo]

+1

[mjtechguy]

+1

[jvlinsta]

+1

[mohittalele]

+1

[ptrmayer]

+1

[tarson96]

++++1

[ptrmayer]

+1

[Luffyzm3D2Y]

+1

[srzer]

+1 desperately need multiple models

[lizhipengpeng]

+1

[amitm02]

+1

[naturomics]

+1

[servient-ashwin]

What is the general thought process or strategy into implementing something like this if this is open for contributions? Does the vLLM team or anyone have a roadmap or idea into it's implementation that someone can pick up?

This would be highly useful. All I can quickly think of is how tensorflow allows to specify memory used something similar to gpu util here on vLLM or using subprocesses within python (although I must admit I do not understand much about this topic in general anyway)

[zr-idol]

+1

[Sk4467]

One work around that I notice vLLM already gives is using Docker containers. Use docker compose to serve multiple models that way two or more models can be served at a time based on your memory capabilities.

[ws-chenc]

+++1

[chen-j-ing]

+1+1

[ArtificialEU]

So i can have a complex vllm setup with batching and all but can't change the current model except spin a new container ? 😅

[pingbowen23]

+1

[ismaslov]

+1

[JoursBleu]

+1

[nathan-weinberg]

+1

[kglmcodes]

+1

[youkaichao]

vLLM does not support multiple models.

for anyone who is interested, please pin up multiple vllm instances, and use routers like litellm https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model to route among vllm instances.