[Feature]: alternate user/assistant message format for async completion calls - openai-compatible endpoint

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate, Groq (100+ LLMs)

https://docs.litellm.ai/docs/

Other

10.8k stars 1.23k forks source link

[Feature]: alternate user/assistant message format for async completion calls - openai-compatible endpoint #2242

Open krrishdholakia opened 5 months ago

krrishdholakia commented 5 months ago

The Feature

we support this for completion, need to support for async completion to work for proxy

Motivation, pitch

user faced issue trying to make calls to mixtral on vllm using us

Twitter / LinkedIn details

No response

abant07 commented 4 months ago

Hey I am interested in this issue, however, I am new to this repo, and I was wondering if you could explain a little more about the issue. I would also like to take a shot at this. Is there anything else I need to know while reproducing the environment other than following the readME?

Thanks.