[Bug]: Let proxy return JSON (for hugging face)

[danielvanmil]

What happened?

How to test:

• Setup proxy with a hugging face model, for example codellama/CodeLlama-34b-Instruct-hf and format="json"
• Use a prompt that let's the model return the json, someting like:

You are a helpful assistent, that only communicates using JSON files.
        The expected output from you has to be: 
                "function_call":  {
            "name": {function_name},
            "args": [],
            "ai_notes": {explanation}
        }

• Use (deprecated) the openAI function calling API (runFunction, etc.)
• the openAI API now:
  • sends JSON to the liteLLM proxy
  • liteLLM proxy understand this and transforms the JSON to a hugging face call
  • Hugging face returns a respons with a "generated_text" property
  • liteLLM proxy doesn't return JSON which the openAI API expects, but something like:

{ 
  content: "....."
  role: "assistent"
}

instead of:

{ 
  function_call: "....."
  role: "assistent"
}

But it could also be possible it don't understand it completely, in that case an answer would be nice.,

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Hey @danielvanmil i can't see if huggingface supports a json mode.

Are you suggesting if a function call is made by the user - dump the response (even if non-json) into the function_call field

[krrishdholakia]

Happy to hop on a call to talk through this - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat