Using gpt-3.5-turbo-0613 from Azure OpenAI services does not work

[aslisabanci]

The bug Using guidance with gpt-3.5-turbo-0613 from Azure OpenAI does not work. Hope I'm missing a silly thing :)

To Reproduce The following code executes fine when using the gpt-3.5-turbo-0613 model from OpenAI but not from an Azure deployment.

import guidance
import json

llm_azure = guidance.llms.OpenAI(
    "gpt-3.5-turbo", 
    api_type="azure",
    api_key="AZURE_OPENAI_API_KEY",
    api_base="AZURE_OPENAI_API_BASE",
    api_version="2023-07-01-preview",
    deployment_id="DEPLOYMENT_NAME_IN_AZURE",
    caching=False,
)
guidance.llm = llm_azure

# WORKS
# llm_openai = guidance.llms.OpenAI(
#    "gpt-3.5-turbo-0613", caching=False, api_key="OPENAI_API_KEY"
#)
# guidance.llm = llm_openai 

print("running guidance version: ", guidance.__version__)

def get_current_weather(location, unit="fahrenheit"):
    """Get the current weather in a given location.

    Parameters
    ----------
    location : string
        The city and state, e.g. San Francisco, CA
    unit : "celsius" or "fahrenheit"
    """
    weather_info = {
        "location": location,
        "temperature": "71",
        "unit": unit,
        "forecast": ["sunny", "windy"],
    }
    return json.dumps(weather_info)

# define a guidance program that uses tools
program = guidance(
    """
{{~#system~}}
You are a helpful assistant.
{{>tool_def functions=functions}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#each range(10)~}}
    {{~#assistant~}}
    {{gen 'answer' max_tokens=50 function_call="auto"}}
    {{~/assistant~}}

    {{#if not callable(answer)}}{{break}}{{/if}}

    {{~#function name=answer.__name__~}}
    {{answer()}}
    {{~/function~}}
{{~/each~}}""",
    silent=False,
)

executed_program = program(
    functions=[
        {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        }
    ],
    get_current_weather=get_current_weather,
)

print(executed_program["answer"])

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): macOS Ventura 13.5
• Guidance Version (guidance.__version__): 0.0.64

Here is the full trace:

running guidance version:  0.0.64
Traceback (most recent call last):
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 109, in run
    await self.visit(self.parse_tree, VariableStack([self.program._variables], self))
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/library/_each.py", line 93, in each
    new_content += await parser.visit(
                   ^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 524, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/library/_assistant.py", line 13, in assistant
    return await role(role_name="assistant", hidden=hidden, _parser_context=_parser_context, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/library/_role.py", line 17, in role
    new_content += await parser.visit(
                   ^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 559, in visit
    visited_children.append(await self.visit(child, variable_stack, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 266, in visit
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 266, in <listcomp>
    visited_children = [await self.visit(child, variable_stack, next_node, next_next_node, prev_node, node, parent_node) for child in node]
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/_program_executor.py", line 379, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/asli.demiroz/workspace/streamlit_chat/streamlit_local_env/lib/python3.11/site-packages/guidance/library/_gen.py", line 169, in gen
    new_text = resp["choices"][0].get("text", "")
               ~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Error in program:  list index out of range

[TomTom101]

Try guidance.llms.AzureOpenAI(…)

[aslisabanci]

Try guidance.llms.AzureOpenAI(…)

@TomTom101 thanks for trying to help - but that throws the following exception

Exception has occurred: NotImplementedError
The AzureOpenAI class has been merged with the OpenAI class for Azure usage. Please use the OpenAI class instead: https://guidance.readthedocs.io/en/latest/example_notebooks/api_examples/llms/OpenAI.html

NotImplementedError: The AzureOpenAI class has been merged with the OpenAI class for Azure usage. Please use the OpenAI class instead: https://guidance.readthedocs.io/en/latest/example_notebooks/api_examples/llms/OpenAI.html

In addition, I'm able to use other model deployments from Azure (like text-davinci-003) successfully with guidance using the same constructor. So the issue seems to be specific to using functions with gpt-3.5-turbo-0613. Azure OpenAI Services made function calling available with their 2023-07-01-preview so I'm guessing that the incompatibility is rooted in there.

[TomTom101]

Sorry, by bad! I am on an older version still.

[seanbenhur]

Are there any updates on this?

[aslisabanci]

@seanbenhur Unfortunately no. The latest release came out in June 20 btw. So, it's been a while this library is not getting attention.

[Keiku]

It seems like the same error as #372. It seems that it is not limited to Azure.

[aslisabanci]

It seems like the same error as #372. It seems that it is not limited to Azure.

@Keiku The problem you're having could be because you're swapping the model with a chat based model (like gpt3.5 or gpt4), i.e. not with a text completion model (like text-davinci). Since chat models are trained with tokens that mark out roles in the prompt, your guidance program should be prompting the model accordingly when you're using chat based ones.

[Keiku]

@aslisabanci Sorry. Do you know how to actually change the code? I couldn't understand your abstract answer.

[antonmosin]

This PR should solve the issue