search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.06k stars 1.39k forks source link

[Feature]: Add response_format for anyscale mistral models #1274

Closed marmikcfc closed 4 weeks ago

marmikcfc commented 8 months ago

The Feature

Anyscale recently added json mode for mistral models mistralai/Mistral-7B-Instruct-v0.1 and mistralai/Mixtral-8x7B-Instruct-v0.1. Adding support for the same can be useful for better modelling of response for downstream task.

Motivation, pitch

We're building infra for AI telephony and we would like to use response format for multiple usecases

  1. For IVR type calls.
  2. Extracting information from the transcripts

As a workaround we're using a forked version but it would be great if we can have this feature in the base project itself -#1275

Twitter / LinkedIn details

https://linkedin.com/in/marmikpandya

krrishdholakia commented 8 months ago

adding the relevant docs here - https://docs.endpoints.anyscale.com/guides/json_mode#supported-models

@marmikcfc thanks, will add this tomorrow.

Would love to learn more about how you're using litellm, attaching my calendly if you have ~10 minutes to chat this/next week - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

krrishdholakia commented 8 months ago

i merged your PR @marmikcfc

I believe for function calling, anyscale seems to have done it differently with the json schema being passed in as part of the json format arg

ishaan-jaff commented 4 weeks ago

anyscale discontinued their API