[Feature Request]: Add function calling capability to LiteLLM

[tituslhy]

Feature Description

LiteLLM is a wrapper over LLMs from non-OpenAI providers to harmonize their APIs to OpenAI's APIs. This ensures harmonization and ease of LLM switchability.

Most LLMs (like Bedrock) have function-calling capabilities, and LiteLLM already handles this on their end. Would it be too difficult to add function-calling capabilities to LlamaIndex's LiteLLM class?

Here's an example of function calling failure once we wrap any LLM into LlamaIndex's LiteLLM class (including OpenAI)

from dotenv import load_dotenv, find_dotenv
from llama_index.llms.litellm import LiteLLM
from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.tools import FunctionTool

_=load_dotenv(find_dotenv()) #load OpenAI API key
llm = LiteLLM(model="gpt-4o-mini")
def add(a: int, b: int) -> int:
    """Returns the sum of 2 integers"""
    return a + b

def multiply(a: int, b: int) -> int:
    """Returns the product of 2 integers"""
    return a * b

add_tool = FunctionTool.from_defaults(fn=add)
multiply_tool = FunctionTool.from_defaults(fn=multiply)
agent_worker = FunctionCallingAgentWorker.from_tools(
    tools = [add_tool, multiply_tool],
    llm = llm,
    verbose = True
)

This throws the error:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[19], line 1
----> 1 agent_worker = FunctionCallingAgentWorker.from_tools(
      2     tools = [add_tool, multiply_tool],
      3     llm = llm,
      4     verbose = True
      5 )

File /opt/anaconda3/envs/llamaindex/lib/python3.12/site-packages/llama_index/core/agent/function_calling/step.py:147, in FunctionCallingAgentWorker.from_tools(cls, tools, tool_retriever, llm, verbose, max_function_calls, allow_parallel_tool_calls, callback_manager, system_prompt, prefix_messages, **kwargs)
    144 tools = tools or []
    146 llm = llm or Settings.llm  # type: ignore
--> 147 assert isinstance(
    148     llm, FunctionCallingLLM
    149 ), "llm must be an instance of FunctionCallingLLM"
    151 if callback_manager is not None:
    152     llm.callback_manager = callback_manager

AssertionError: llm must be an instance of FunctionCallingLLM

Funnily enough LlamaIndex has code to identify that this LLM within LiteLLM is a function calling LLM in lines 160-169 of llama_index/llama

 @property
    def metadata(self) -> LLMMetadata:
        return LLMMetadata(
            context_window=openai_modelname_to_contextsize(self._get_model_name()),
            num_output=self.max_tokens or -1,
            is_chat_model=True,
            is_function_calling_model=is_function_calling_model(self._get_model_name()),
            model_name=self.model,
        )

printing llm.metadata returns:

LLMMetadata(context_window=16384, num_output=-1, is_chat_model=True, is_function_calling_model=True, model_name='gpt-4o-mini', system_role=<MessageRole.SYSTEM: 'system'>)

As we can see, is_function_calling_model = True. But the reason LiteLLM function calling still returns an error is because the LiteLLM class inherits from "LLM" instead of "FunctionCallingLLM". I sense that this is not a trivial fix.

Reason

I could use LiteLLM's library directly but it would be great to use LlamaIndex's capabilities!

Value of Feature

LiteLLM is one of the commonly used libraries to harmonize LLM provider APIs to the OpenAI API, and the library continues to be used despite more libraries being harmonized to OpenAI APIs. LiteLLM continues to update their offerings to remain current across LLM providers.

[logan-markewich]

@tituslhy it would just have to extend the FunctionCallingLLM base class and implement a few extra functions -- I welcome a PR to add this :)

[tituslhy]

Oops I cheated. I just used LlamaIndex's ReAct workflow cookbook recipe (https://docs.llamaindex.ai/en/stable/examples/workflow/react_agent/) and used LiteLLM as my llm of interest HAHA! I also just initiated ReAct agent from tools and used LiteLLM. Heh.

I did experiment a little with inheriting from FunctionCallingLLM but I found this way harder - when I called Gemini from LIteLLM it also had tool calling json errors though it fit LiteLLM's schema perfectly. I think that it's really difficult to cater to every single LLM service provider.