[FR] Support JSON mode

[NightMachinery]

Some models expose a JSON-mode API where they output correct JSON. There are some related APIs such as function calling (tool use). These are useful for programmatic use, as we need in an IDE like emacs.

Here is an example:

• JSON Mode

  
  import os
  import json
  import openai
  from pydantic import BaseModel, Field

Create client

client = openai.OpenAI( base_url="https://api.together.xyz/v1", api_key=os.environ["TOGETHER_API_KEY"], )

Define the schema for the output.

class User(BaseModel): name: str = Field(description="user name") address: str = Field(description="address")

Call the LLM with the JSON schema

chat_completion = client.chat.completions.create( model="mistralai/Mixtral-8x7B-Instruct-v0.1", response_format={"type": "json_object", "schema": User.model_json_schema()}, messages=[ { "role": "system", "content": "You are a helpful assistant that answers in JSON.", }, { "role": "user", "content": "Create a user named Alice, who lives in 42, Wonderland Avenue.", }, ], )

created_user = json.loads(chat_completion.choices[0].message.content) print(json.dumps(created_user, indent=2))

""" { "address": "42, Wonderland Avenue", "name": "Alice" } """

[ahyatt]

We already support function calling. Is that not enough for the use-cases where you need correct output?

Additionally, something on my plate is to release a way to do more advanced things, like parse our JSON from any response, and possibly tie that into a state machine where we might go back to the LLM if it doesn't produce valid JSON. But recent models are quite good, and generally don't need anything but code to parse out JSON from a response.

[NightMachinery]

Ah, I didn't RTFM. Sorry. I thought function calling wasn't supported.

I'll close the issue for now to do more testing and see if JSON mode needs specific support.

For reference:

• https://github.com/ahyatt/llm#function-calling