[Bug]: QueryPlanTool is failing when trying to use too many tools

[dhruv-clootrack]

Bug Description

I'm trying to use QueryPlanTool for task of planning and executing the custom function-as-tool created, for 3 tools it was working fine but when i tried to increase the no of tools it failed with exception from OpenAI

BadRequestError: Error code: 400 - {'error': {'message': "'        This is a query plan tool that takes in a list of tools and executes a query plan over these tools to answer a query. The query plan is a DAG of query nodes.\\n\\nGiven a list of tool names and the query plan schema, you can choose to generate a query plan to answer a question.\\n\\nThe tool names and descriptions are as follows:\\n\\n\\n\\n        Tool Name: sept_2022\\nTool Description: Provides information about Uber quarterly financials ending September 2022 \\n\\nTool Name: june_2022\\nTool Description: Provides information about Uber quarterly financials ending June 2022 \\n\\nTool Name: march_2022\\nTool Description: Provides information about Uber quarterly financials ending March 2022 \\n\\nTool Name: long_description\\nTool Description: Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber quarterly financials ending March 2022Provides information about Uber ...

I have identified the issue with QueryPlanTool, while sending the request to OpenAI, it convert all the function-names, function-docs into a string and send in the description of tool named query_plan_tool, but the description became very large where as description field has max-size-limit of 1024 chars on openAI

Version

llama-index==0.10.6

Steps to Reproduce

Use colab-notebook as base: link

Add another cell with code:

query_tool_long_description = QueryEngineTool.from_defaults(
    query_engine=march_engine,
    name="long_description",
    description=(
        f"Provides information about Uber quarterly financials ending March"
        f" 2022"*100
    ),
)

query_plan_tool_with_issue = QueryPlanTool.from_defaults(
    query_engine_tools=[query_tool_sept, query_tool_june, query_tool_march, query_tool_long_description],
    response_synthesizer=response_synthesizer,
)

agent_with_issue = OpenAIAgent.from_tools(
    [query_plan_tool_with_issue],
    max_function_calls=10,
    llm=OpenAI(temperature=0, model="gpt-4-0613"),
    verbose=True,
)

agent_with_issue.query("What were the risk factors in sept 2022 vs june 2022?")

and run

Relevant Logs/Tracbacks

---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
Cell In[234], line 1
----> 1 response = agent.query(
      2     "Analyze customer issues related to packaging and finally do SWOT Analysis"
      3 )

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/base/base_query_engine.py:40, in BaseQueryEngine.query(self, str_or_query_bundle)
     38 if isinstance(str_or_query_bundle, str):
     39     str_or_query_bundle = QueryBundle(str_or_query_bundle)
---> 40 return self._query(str_or_query_bundle)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/callbacks/utils.py:41, in trace_method.<locals>.decorator.<locals>.wrapper(self, *args, **kwargs)
     39 callback_manager = cast(CallbackManager, callback_manager)
     40 with callback_manager.as_trace(trace_id):
---> 41     return func(self, *args, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/agent/types.py:40, in BaseAgent._query(self, query_bundle)
     38 @trace_method("query")
     39 def _query(self, query_bundle: QueryBundle) -> RESPONSE_TYPE:
---> 40     agent_response = self.chat(
     41         query_bundle.query_str,
     42         chat_history=[],
     43     )
     44     return Response(
     45         response=str(agent_response), source_nodes=agent_response.source_nodes
     46     )

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/callbacks/utils.py:41, in trace_method.<locals>.decorator.<locals>.wrapper(self, *args, **kwargs)
     39 callback_manager = cast(CallbackManager, callback_manager)
     40 with callback_manager.as_trace(trace_id):
---> 41     return func(self, *args, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/agent/runner/base.py:575, in AgentRunner.chat(self, message, chat_history, tool_choice)
    570     tool_choice = self.default_tool_choice
    571 with self.callback_manager.event(
    572     CBEventType.AGENT_STEP,
    573     payload={EventPayload.MESSAGES: [message]},
    574 ) as e:
--> 575     chat_response = self._chat(
    576         message, chat_history, tool_choice, mode=ChatResponseMode.WAIT
    577     )
    578     assert isinstance(chat_response, AgentChatResponse)
    579     e.on_end(payload={EventPayload.RESPONSE: chat_response})

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/agent/runner/base.py:520, in AgentRunner._chat(self, message, chat_history, tool_choice, mode)
    517 result_output = None
    518 while True:
    519     # pass step queue in as argument, assume step executor is stateless
--> 520     cur_step_output = self._run_step(
    521         task.task_id, mode=mode, tool_choice=tool_choice
    522     )
    524     if cur_step_output.is_last:
    525         result_output = cur_step_output

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/agent/runner/base.py:372, in AgentRunner._run_step(self, task_id, step, input, mode, **kwargs)
    368 # TODO: figure out if you can dynamically swap in different step executors
    369 # not clear when you would do that by theoretically possible
    371 if mode == ChatResponseMode.WAIT:
--> 372     cur_step_output = self.agent_worker.run_step(step, task, **kwargs)
    373 elif mode == ChatResponseMode.STREAM:
    374     cur_step_output = self.agent_worker.stream_step(step, task, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/callbacks/utils.py:41, in trace_method.<locals>.decorator.<locals>.wrapper(self, *args, **kwargs)
     39 callback_manager = cast(CallbackManager, callback_manager)
     40 with callback_manager.as_trace(trace_id):
---> 41     return func(self, *args, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/agent/openai/step.py:572, in OpenAIAgentWorker.run_step(self, step, task, **kwargs)
    570 """Run step."""
    571 tool_choice = kwargs.get("tool_choice", "auto")
--> 572 return self._run_step(
    573     step, task, mode=ChatResponseMode.WAIT, tool_choice=tool_choice
    574 )

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/agent/openai/step.py:447, in OpenAIAgentWorker._run_step(self, step, task, mode, tool_choice)
    443 openai_tools = [tool.metadata.to_openai_tool() for tool in tools]
    445 llm_chat_kwargs = self._get_llm_chat_kwargs(task, openai_tools, tool_choice)
--> 447 agent_chat_response = self._get_agent_response(
    448     task, mode=mode, **llm_chat_kwargs
    449 )
    451 # TODO: implement _should_continue
    452 latest_tool_calls = self.get_latest_tool_calls(task) or []

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/agent/openai/step.py:321, in OpenAIAgentWorker._get_agent_response(self, task, mode, **llm_chat_kwargs)
    317 def _get_agent_response(
    318     self, task: Task, mode: ChatResponseMode, **llm_chat_kwargs: Any
    319 ) -> AGENT_CHAT_RESPONSE_TYPE:
    320     if mode == ChatResponseMode.WAIT:
--> 321         chat_response: ChatResponse = self._llm.chat(**llm_chat_kwargs)
    322         return self._process_message(task, chat_response)
    323     elif mode == ChatResponseMode.STREAM:

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py:93, in llm_chat_callback.<locals>.wrap.<locals>.wrapped_llm_chat(_self, messages, **kwargs)
     84 with wrapper_logic(_self) as callback_manager:
     85     event_id = callback_manager.on_event_start(
     86         CBEventType.LLM,
     87         payload={
   (...)
     91         },
     92     )
---> 93     f_return_val = f(_self, messages, **kwargs)
     95     if isinstance(f_return_val, Generator):
     96         # intercept the generator and add a callback to the end
     97         def wrapped_gen() -> ChatResponseGen:

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/llms/openai/base.py:237, in OpenAI.chat(self, messages, **kwargs)
    235 else:
    236     chat_fn = completion_to_chat_decorator(self._complete)
--> 237 return chat_fn(messages, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/llama_index/llms/openai/base.py:296, in OpenAI._chat(self, messages, **kwargs)
    294 client = self._get_client()
    295 message_dicts = to_openai_message_dicts(messages)
--> 296 response = client.chat.completions.create(
    297     messages=message_dicts,
    298     stream=False,
    299     **self._get_model_kwargs(**kwargs),
    300 )
    301 openai_message = response.choices[0].message
    302 message = from_openai_message(openai_message)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/openai/_utils/_utils.py:275, in required_args.<locals>.inner.<locals>.wrapper(*args, **kwargs)
    273             msg = f"Missing required argument: {quote(missing[0])}"
    274     raise TypeError(msg)
--> 275 return func(*args, **kwargs)

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/openai/resources/chat/completions.py:663, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
    611 @required_args(["messages", "model"], ["messages", "model", "stream"])
    612 def create(
    613     self,
   (...)
    661     timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
    662 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
--> 663     return self._post(
    664         "/chat/completions",
    665         body=maybe_transform(
    666             {
    667                 "messages": messages,
    668                 "model": model,
    669                 "frequency_penalty": frequency_penalty,
    670                 "function_call": function_call,
    671                 "functions": functions,
    672                 "logit_bias": logit_bias,
    673                 "logprobs": logprobs,
    674                 "max_tokens": max_tokens,
    675                 "n": n,
    676                 "presence_penalty": presence_penalty,
    677                 "response_format": response_format,
    678                 "seed": seed,
    679                 "stop": stop,
    680                 "stream": stream,
    681                 "temperature": temperature,
    682                 "tool_choice": tool_choice,
    683                 "tools": tools,
    684                 "top_logprobs": top_logprobs,
    685                 "top_p": top_p,
    686                 "user": user,
    687             },
    688             completion_create_params.CompletionCreateParams,
    689         ),
    690         options=make_request_options(
    691             extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
    692         ),
    693         cast_to=ChatCompletion,
    694         stream=stream or False,
    695         stream_cls=Stream[ChatCompletionChunk],
    696     )

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/openai/_base_client.py:1200, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1186 def post(
   1187     self,
   1188     path: str,
   (...)
   1195     stream_cls: type[_StreamT] | None = None,
   1196 ) -> ResponseT | _StreamT:
   1197     opts = FinalRequestOptions.construct(
   1198         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1199     )
-> 1200     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/openai/_base_client.py:889, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    880 def request(
    881     self,
    882     cast_to: Type[ResponseT],
   (...)
    887     stream_cls: type[_StreamT] | None = None,
    888 ) -> ResponseT | _StreamT:
--> 889     return self._request(
    890         cast_to=cast_to,
    891         options=options,
    892         stream=stream,
    893         stream_cls=stream_cls,
    894         remaining_retries=remaining_retries,
    895     )

File ~/Desktop/Work/lamma-index/.venv/lib/python3.10/site-packages/openai/_base_client.py:980, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
    977         err.response.read()
    979     log.debug("Re-raising status error")
--> 980     raise self._make_status_error_from_response(err.response) from None
    982 return self._process_response(
    983     cast_to=cast_to,
    984     options=options,
   (...)
    987     stream_cls=stream_cls,
    988 )

BadRequestError: Error code: 400 - {'error': {'message': '"        This is a query plan tool that takes in a list of tools and executes a query plan over these tools to answer a query. The query plan is a DAG of query nodes.\\n\\nGiven a list of tool names and the query plan schema, you can choose to generate a query plan to answer a question.\\n\\nThe tool names and descriptions

[dosubot[bot]]

🤖

It's great that you've identified the potential cause of the issue with the description field exceeding the size limit for the OpenAI request. Have you considered any potential solutions or workarounds for this issue? Your insights could be valuable in finding a resolution. If you're able to propose a fix or a workaround, we'd encourage you to consider creating a pull request to contribute your solution to the repository. Your contribution could help improve the tool for everyone.

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

Theres not really much we can do about this? Indeed, OpenAI has a limit for the lengths of tools/descriptions in their API

[logan-markewich]

Need to shorten the tool names/descriptions I think, would be the main course of action?

[dhruv-clootrack]

Yes that is the only way around with OpenAI Limit

[dhruv-clootrack]

Can we try solution like:

Instead of passing all the function-names, function-docs into a description, can we create each function into a separate tool and pass them along with query_plan_tool?

I haven't tried it.

[dosubot[bot]]

Hi, @dhruv-clootrack,

I'm helping the LlamaIndex team manage their backlog and am marking this issue as stale. The issue was raised by you regarding the QueryPlanTool failing due to a BadRequestError from OpenAI. The cause was identified as the description of the tool exceeding the max-size-limit of 1024 characters. You suggested a potential solution of creating each function into a separate tool and passing them along with query_plan_tool to address the issue. Other contributors acknowledged the limitation imposed by OpenAI and discussed the need to shorten tool names/descriptions as the main course of action.

Could you please confirm if this issue is still relevant to the latest version of the LlamaIndex repository? If it is, please let the LlamaIndex team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

I