[Feature] Support TRI-ML/prismatic-vlms - Githubissues

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.

https://sglang.readthedocs.io/en/latest/

Apache License 2.0

5.11k stars 356 forks source link

[Feature] Support TRI-ML/prismatic-vlms #1129

Open Depetrol opened 4 weeks ago

Depetrol commented 4 weeks ago

Checklist

[X] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
[X] 2. Please use English, otherwise it will be closed.

Motivation

I'm trying to speed up inference for new VLM models on huggingface: https://huggingface.co/TRI-ML/prismatic-vlms/tree/main. I'm wondering if there are additional documentation on how to adapt new models?

Related resources

The model I'm trying to adapt is detailed here: https://arxiv.org/pdf/2402.07865.

merrymercy commented 2 weeks ago

Contributions are welcome! Please refer to https://github.com/sgl-project/sglang/blob/main/docs/en/model_support.md and the existing llava implementation