Support for vLLM and TRT-LLM running in OpenAI compatible mode

[vecorro]

Is your feature request related to a problem? Please describe. I'd like to be able to run vLLM emulating the OpenAI compatible API to use vLLM as a drop-in replacement of ChatGPT.

Describe the solution you'd like I'd like Triton allow me run vLLM as indicated in vLLM documentation

Example:

python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

Describe alternatives you've considered It is possible to use the REST API, however, for developers already leveraging OpenAI and serving open-source LLMs using the OpenAI API would allow a faster path for replacement of OpenAI

[krishung5]

Thanks for submitting feature request! CC @nnshah1 on the request for starting server with the OpenAI compatible API.

For the client side, we have introduced the generate endpoint. which is an OpenAI-like endpoint support and can improve Triton adoption for LLM use cases.

[nnshah1]

@vecorro It's a bit challenging as the OpenAI API is a moving target and already the completion api is legacy and out of date with many applications. As triton as an inference server serves many types of inference we've focused on providing ways to customize endpoints also with the expectation that many actual deployments would use an additional api gateway to translate from a service api to the inference api (for example langchain).

that being said - your perspective is important. Is the legacy completion api enough or would we need to support the newer apis as well? Do you think integration with a project like langchain or other llm projects would be a viable approach or would triton need to provide the interface directly?

[chymian]

@nnshah1

that being said - your perspective is important. Is the legacy completion api enough or would we need to support the newer apis as well? Do you think integration with a project like langchain or other llm projects would be a viable approach or would triton need to provide the interface directly?

The key here, to be usefull to the community, is actuality & compatibility. every other often used tool, like LiteLLM (middlware/API-Proxy) or loaders, like ooba's TGI, vLLM, FastChat are on the run to implement functions and to pick up with v1, since this is essential for local LLMs to be a replacement for the costly openai-stuff. There are a few very popular projects which use the very latest features heavly, like Mircosofts Autogen and MemGPT.

IMHO: I have been testing all these and just had a peek into triton, but already assume, it would be the much supirior solution of running multiple LLMs locally - IF there would be an out of the box compatibility with the openAI-API. Maybe teaming up with https://litellm.ai/ is a quick solution.

[ishaan-jaff]

@chymian @nnshah1 im the litellm maintainer - what do you need from us ?

[vecorro]

I agree with @nnshah1; the key is actuality and compatibility, and I would add ease of use. For experimentation/ded purposes, vLLM is excellent as it allows you to try an LLM without the complexity and time required by TensorRT-LLM. Making the existing vLLM implementation of the legacy OpenAI API can be good enough; meanwhile, vLLM evolves to support newer versions of the OpenAI API. I'm on the VMware team working with NVIDIA on the private AI initiative. From conversations with customers looking for ways to run LLMs on-premises, they are getting started on that journey, so a quick drop-in replacement of OpenAI's models would facilitate things for them. Thanks all!

[nnshah1]

@npuichigo has mentioned an integration that could be useful here:

https://github.com/NVIDIA/TensorRT-LLM/discussions/591

[npuichigo]

@npuichigo has mentioned an integration that could be useful here:

NVIDIA/TensorRT-LLM#591

https://github.com/npuichigo/openai_trtllm provides an OpenAI-like API for trtllm triton backend, but I think vllm in triton would be something alike.

[dyastremsky]

Closing issue due to inactivity. Please reopen if you would like to follow up with this issue.

[BodhiHu]

Hello,

Seems it's supporting vllm now: https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton

But can we use the vllm OpenAI APIs ?

Thanks a lot

[nnshah1]

We don't currently support it directly - but are still thinking through the best ways to add a compatible API in a way that we can maintain.

[BodhiHu]

Thank you for the clearance~~ though I think we could simply adapte the vLLM vllm.entrypoints.openai.api_server to the triton http endpoint.

[panpan0000]

is there any workaround so far ?

Thank you for the clearance~~ though I think we could simply adapte the vLLM vllm.entrypoints.openai.api_server to the triton http endpoint.

[nnshah1]

@chymian @nnshah1 im the litellm maintainer - what do you need from us ?

@ishaan-jaff

Is there a guide on adding an llm provider to litellm or a proscribed starting point / skeleton?

[anubhav-agrawal-mu-sigma]

Is this planned for any specific release?

[nnshah1]

@anubhav-agrawal-mu-sigma - we are currently planning a tutorial showcasing how to create an open ai compatible triton server using triton's in-process python api - that tutorial is planned for September time frame.

[catle2aurecon]

@nnshah1 : I am interested in the mentioned tutorial, please let me know when you have it online. Many thanks !

[agladboy]

@nnshah1 I’m excited to see the tutorial! Would you please to let me know when it will be available? Thanks in advance!

[BrandesDenis]

@nnshah1 I'm interested too!

[nnshah1]

You can follow along the current preview PR here:

https://github.com/triton-inference-server/server/pull/7561

Some things may change- so please consider this as BETA until we finalize a few of the internal structure - but this will be the basis.