Unable to generate JSON with a chat model

[dokluch]

The bug Chat models seemingly can't generate JSON as output.

To Reproduce I am using the same schema as with text-davinci, but wrapping it with the assistant role:

{{#assistant~}}
```json
{
    "title": "{{gen 'title'}}",
}```
{{~/assistant}}

The error I get is: AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.

It works without JSON, though.

System info (please complete the following information):

• OS: macOS
• Guidance Version: 0.0.61

[Akhilkapoor12]

facing the same issue with text-davinci as well, it says

"""openAI dose not support creating assert not pattern, "The OpenAI API does not support Guidance pattern controls! Please either switch to an endpoint that does, or don't use the pattern argument to gen." AssertionError: The OpenAI API does not support Guidance pattern controls! Please either switch to an endpoint that does, or don't use the pattern argument to gen."""

[KMontag42]

This pattern of generation is only supported when using guidance.llms.Transformers

see the notebook

[AlvinNg89]

facing the same issue with text-davinci as well, it says

"""openAI dose not support creating assert not pattern, "The OpenAI API does not support Guidance pattern controls! Please either switch to an endpoint that does, or don't use the pattern argument to gen." AssertionError: The OpenAI API does not support Guidance pattern controls! Please either switch to an endpoint that does, or don't use the pattern argument to gen."""

For this case, I had to remove the pattern argument (as suggested in the error msg) in the handlebar to generate the json output. Seems like we have to rely on OpenAI model to deduce the right pattern for each field.

[imlasky]

As a workaround, I've followed the prompt from here with a parser function (below) to just get the json object, just in case there are still explanations around the JSON. This seems to work pretty reliably.

def get_json(text):
    pattern = regex.compile(r'\{(?:[^{}]|(?R))*\}')

    return pattern.findall(text)[0]

[imlasky]

Looks like OpenAI released function calling yesterday in their latest gpt-3.5-turbo-0613 which allows native JSON rendering. Here's a good writeup about using it.