search

deepjavalibrary / djl-serving

A universal scalable machine learning model deployment solution
Apache License 2.0
196 stars 64 forks source link

[python] support multimodal models openai api in vllm #2147

Closed sindhuvahinis closed 2 months ago

sindhuvahinis commented 3 months ago

Description

Waiting for vllm upgrade 0.5.1 in our lmi.Dockerfile

This needs vLLM 0.5.1 version, but our container is not updated yet.

Testing:

  1. Manually tested in my EC2 with vllm 0.5.0.post1 installed
  2. Added integration test for llava-chat
  3. Added unit test case for parsing the chat template with vision
sindhuvahinis commented 2 months ago

Waiting for vllm upgrade 0.5.1 in our lmi.Dockerfile