Closed XBeg9 closed 7 months ago
I have tried to use ray-llm, but it doesn't work out of the box with what vllm suppose to provide. Is there any changes that ray-llm provides to the API endpoint while inference the model?
Here is comment on issue to vllm repo that was solved long time ago: https://github.com/vllm-project/vllm/pull/323#issuecomment-1819862501
you can use the ChatAnyscale module https://api.python.langchain.com/en/latest/chat_models/langchain.chat_models.anyscale.ChatAnyscale.html with the api endpoint set to your endpoint
I have tried to use ray-llm, but it doesn't work out of the box with what vllm suppose to provide. Is there any changes that ray-llm provides to the API endpoint while inference the model?
Here is comment on issue to vllm repo that was solved long time ago: https://github.com/vllm-project/vllm/pull/323#issuecomment-1819862501