[Usage]: Deploying multimodal retrieval models

[sky-2002]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I want to run inference of ColPali. I don't know how to integrate it with vllm. It used PaliGemma which is there in vLLM , but it also loads some adapters. Please let me know if it can be used right-away, or if any changes need to be made, let me know, I am happy to contribute.

Before submitting a new issue...

• [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

LoRA adapters are not supported yet for multi-modal models. See #7199.

[sky-2002]

hey @DarkLight1337 , if not the whole model, only this part and inference part cane be done with vLLM right?

[DarkLight1337]

Currently yes, you are welcome to extend vLLM with new embedding models though!