search

rh-aiservices-bu / llm-on-openshift

Resources, demos, recipes,... to work with LLMs on OpenShift with OpenShift AI or Open Data Hub.
Apache License 2.0
97 stars 102 forks source link

Granite support in vLLM containers #71

Closed tombenninger closed 4 months ago

tombenninger commented 5 months ago

It looks like full Granite support was added to vLLM in v0.4.3, and there are some enhancements coming in the 'vtest' release.

Currently, I only see v0.4.2 available in Quay: https://quay.io/rh-aiservices-bu/vllm-openai-ubi9:0.4.2

Coud the vllm-openai-ubi9 image be rebuilt with vLLM v0.4.3?

Does it make sense to also provide vtest release images?

guimou commented 4 months ago

As we are now releasing our own vLLM images as part of ODH/RHOAI, I am not sure I will continue updating the ones here. If you want you can use this one: https://github.com/opendatahub-io/vllm/releases/tag/2024.07.10 which more or less matches to v0.5. Built images are available here: https://quay.io/repository/modh/vllm

guimou commented 4 months ago

I will close this for now, while thinking about what I will do with the vllm images.