[Bug]: BaseChatEngine.chat() returns None if OpenAi is used and function_call="query_engine_tool"

[NoahMoritz]

Bug Description

Hello all,

I've tried a few different scenarios but the basic underlying issue seems to be, that whenever I use a Cheat Engine with OpenAi, the return value from all chat functions is None. Here a sample code where the issues exists:

import os
os.environ["OPENAI_API_KEY"] = '----------------------------------'

from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.chat_engine.types import ChatMode
from llama_index.llms import OpenAI

service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo-0613"))
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents, service_context=service_context)

chat_engine = index.as_chat_engine(chat_mode=ChatMode.OPENAI, Verbose=True)

while True:
    user_msg = input("User: ")
    print(chat_engine.chat(user_msg, function_call="query_engine_tool"))

If I remove the function_call, I get a response (which then does not use the index, no matter what the question is). If I leave it in, (I've tried this in another scenario) then it does ask questions to the query engine (so it's being used and the results are good), but the result is still None.

What I have also tried:

• using GPT-4
• using an old version of llama-index
• trying different ways to create the chat engine
• using different methods (async chat, ...)
• using a different venv

I don't know if I'm doing something wrong or if this is bug. Since (at least for me) it is reproducible all the time in different scenarios, I assume it is a bug. If I'm wrong, help would be appreciated.

Best

Version

0.8.5.post2

Steps to Reproduce

Use the code, but your API Key in and but some random Textfile into a folder called "data". Then ask questions about this file.

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

🤖

Hello,

Thank you for bringing this issue to our attention. Based on the context provided, it seems like the BaseChatEngine.chat() method in the LlamaIndex v0.8.5.post2 could return None when using OpenAI and function_call="query_engine_tool" due to several reasons:

1. The function call does not exist in the list of tools: If the function call specified in the function_call parameter does not exist in the list of tools, the get_function_by_name() method will raise a ValueError and the chat method will return None.

2. The function call does not return any output: If the function call does not return any output, the call_function() or acall_function() method will return a ChatMessage with None content, which will be added to the memory and returned by the chat method.

3. The function call exceeds the maximum number of function calls: The chat method has a loop that continues until no more function calls are available or the maximum number of function calls is reached. If the maximum number of function calls is reached, the _should_continue() method will return False and the chat method will break the loop and return the last agent chat response, which could be None if the last function call did not return any output.

4. The function call does not support the function calling API: In the context shared, the from_defaults() method in the OpenAIPydanticProgram class raises a ValueError if the model does not support the function calling API. This could cause the chat() method to return None.

5. The function call does not exist in the AI message's additional kwargs: The __call__() and acall() methods in the OpenAIPydanticProgram class raise a ValueError if the function call does not exist in the AI message's additional kwargs. This could cause the chat() method to return None.

6. The QueryEngineTool does not return a response: In the context shared, the call() and acall() methods in the QueryEngineTool class return a ToolOutput with the content set to the string representation of the response from the query engine. If the query engine does not return a response or returns a response that when converted to a string is None, the chat() method will return None.

To fix this issue, you should ensure that the function call exists in the list of tools, the function call returns some output, the number of function calls does not exceed the maximum number of function calls, the function call supports the function calling API, the function call exists in the AI message's additional kwargs, and the QueryEngineTool returns a response.

I hope this helps! If you have any other questions or need further clarification, please let me know.

Best, Dosu

Sources

• [Question]: Support for function call api for OpenAI
• llama_index/agent/openai_agent.py
• llama_index/program/openai_program.py
• llama_index/tools/query_engine.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[NoahMoritz]

@dosu-bot You are right, the return object itself is not None. The return object is a AgentChatResponse. But this object looks like this:

{
  "response": None,
  "source_nodes": [0,1,2,3,4,5], # 11 Source Nodes
  "sources": [0,1,2,3,4,5] # 5 Sources
}

But the issue is therefore still, that the response (and I mean the text response) is None.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed description of your issue. Based on the context provided, there could be several reasons why the BaseChatEngine.chat() method is returning an AgentChatResponse object with a None response when using OpenAI and function_call="query_engine_tool".

1. The function_call argument might not be correctly formatted or does not match the expected format. The function_call argument should be a dictionary with the name and arguments keys. If it's a string, it might not be correctly parsed.

2. The function_call argument does not correspond to a valid function in the OpenAI model. If the function specified in function_call does not exist or is not supported by the OpenAI model, the method might return a None response.

3. The OpenAI model does not support the function calling API. The code checks if the model supports the function calling API with llm.metadata.is_function_calling_model. If the model does not support this API, a ValueError is raised.

4. In your code, the OpenAI class is used, which has its own _chat(), _achat(), stream_chat(), and astream_chat() methods. If these methods are being used instead of the BaseChatEngine, it could be the source of the issue. The _chat(), _achat(), stream_chat(), and astream_chat() methods in OpenAI initialize the chat, get the function calls, and process the messages. If any of these steps are not functioning correctly, it could result in a None response.

To resolve this issue, you can:

1. Ensure that the function_call argument is correctly formatted and corresponds to a valid function in the OpenAI model.

2. Use an OpenAI model that supports the function calling API.

3. If using OpenAI, ensure that the chat initialization, function call retrieval, and message processing are functioning correctly.

Unfortunately, I wasn't able to find the expected format for the function_call argument in the BaseChatEngine.chat() method within the repository. It's possible that this information is available elsewhere or I might have missed it.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Best, Dosu

Sources

• [Question]: Support for function call api for OpenAI
• Query Response Returning NONE (Custom LLM)
• Not getting an answer
• llama_index/program/openai_program.py
• llama_index/chat_engine/condense_question.py
• llama_index/agent/openai_agent.py
• llama_index/llms/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[NoahMoritz]

@dosu-bot Okay, I analyzed the issue based on your help. I think, what you said might not be the issue. When I look into the chat history, I can see that the there is one "USER" message, 7 "ASSISTANT" messages and 6 "FUNCTION" messages. The last message is a assistant message which contains the content None. But it contains one function call. The reason why I think this might be a problem is, that the _max_function_calls is defaulted to 5. So the LLM tries to call a function, but the limit is exceeded and therefore the content is returned.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your issue. Based on the context provided, it seems like your suspicion about the _max_function_calls defaulting to 5 could be correct. If the number of function calls exceeds this limit, the _should_continue() function will return False and the while loop in the _chat() method breaks, potentially leading to a None response.

Here's the relevant code snippet from the BaseChatEngine class:

def _should_continue(
    self, function_call: Optional[dict], n_function_calls: int
) -> bool:
    if n_function_calls > self._max_function_calls:
        return False
    if not function_call:
        return False
    return True

This function checks if the number of function calls has exceeded the maximum limit (_max_function_calls) or if there is no function call. If either of these conditions is true, the function returns False.

As for the function_call parameter in the BaseChatEngine.chat() method, it's used to specify the function to be called during the chat. If it's set to "auto", the function to be called is determined automatically based on the chat history and the current message. If it's a string, it's assumed to be the name of the function to be called. If it's a dictionary, it's assumed to be the function call itself, with the keys being the function name and the values being the arguments.

I also found a similar issue in the LlamaIndex repository: [Question]: Support for function call api for OpenAI . The suggested solution there is to access the function calling API using the low-level LLM object as demonstrated in the OpenAIAgent.

I hope this helps! If you have any other questions or need further clarification, please let me know.

Sources

• [Question]: Support for function call api for OpenAI
• llama_index/agent/openai_agent.py
• llama_index/chat_engine/types.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[NoahMoritz]

@dosu-bot Okay, i think I now found the exact cause. The max function calls loop is the problem, because the call to OpenAi will always set the function_call parameter in the API (see Open Ai API documentation) to whatever is defined. The model will only finish with a user message if it is set to "none" or "auto". This implementation however (when it reaches the limit) just returns. Instead one of the following things should happen:

• Either the first request only should contain the request to call the function
• After the limit is reached there should be a last request with "none" in the function_call parameter of the OpenAi API, because otherwise there will be no human readable error message.

[dosubot[bot]]

🤖

Hello,

Thank you for your detailed explanation of the issue you're facing. It seems like you've already done a great deal of troubleshooting.

Based on the context provided, it appears that the issue arises when the _max_function_calls limit is exceeded. In the BaseOpenAIAgent class, the _should_continue method checks if the number of function calls (n_function_calls) has exceeded the _max_function_calls limit. If it has, the method returns False, and the loop in the _chat and _achat methods breaks, which could result in a None response.

To resolve this issue, you can increase the max_function_calls parameter when initializing the OpenAIAgent or RetrieverOpenAIAgent class. This parameter defaults to DEFAULT_MAX_FUNCTION_CALLS, which is set to 5. You can increase this limit according to your requirements.

Here's an example of how to do this:

agent = OpenAIAgent(
    tools=your_tools,
    llm=your_llm,
    memory=your_memory,
    prefix_messages=your_prefix_messages,
    verbose=your_verbose,
    max_function_calls=10,  # Increase this limit as needed
    callback_manager=your_callback_manager,
)

Please replace your_tools, your_llm, your_memory, your_prefix_messages, and your_callback_manager with your actual variables.

Remember that increasing the limit may have implications on the performance and resource usage of your application, so it's important to find a balance that suits your specific needs.

Regarding the function_call parameter in the BaseChatEngine.chat() method, I wasn't able to find specific information within the repository. However, based on your description, it seems like it's used to determine the type of function call to make to the OpenAI API.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Best, Dosu

Sources

• [Question]: Support for function call api for OpenAI
• llama_index/agent/openai_agent.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@NoahMoritz yea I just ran into this today too. It's weird that it USED to work according to the notebook showcasing the feature.

I agree, the function call should only be forced for the first iteration, then disregarded. I can try and fix this later today (or feel free to take a stab at it as well!)