[Feature Request]: Function-calling query engines

[SlapDrone]

Feature Description

Hey folks,

Is there yet implemented a way to integrate function calling APIs with a good old fashioned query engine?

It would be amazing if we had (have?) something like the OpenAI Pydantic Program, but where the function calling + pydantic native output templating was available for complex response synthesizers.

Reason

I don't think there's anything stopping this. In fact it may be in already and just not well documented since things are moving so fast.

I'm currently working with a use-case where I have a query engine that's refining an answer over a large corpus, where the answer is a complex JSON containing an array of objects (currently using the vanilla pydantic + output parser approach). The formatting instructions eat up a huge chunk of my token budget, so this would be a godsend.

Value of Feature

No response

[logan-markewich]

This isn't implemented yet, but I think it makes sense to have at some point! Something like a PydanticQueryEngine that extracts data for a certain pydantic class as it goes? I guess the exact details would need to be worked out.

Like for example, do I define a class like Person(name, number) and the query engine returns a list of those?

[SlapDrone]

Maybe can sketch something out to see if it makes sense. The current single-prompt version from the docs is:

prompt_template_str = """\
Generate an example album, with an artist and a list of songs. \
Using the movie {movie_name} as inspiration.\
"""
program = OpenAIPydanticProgram.from_defaults(
    output_cls=Album,
    prompt_template_str=prompt_template_str,
    verbose=True,
)
output = program(movie_name="The Shining")

Perhaps we could aim to mirror this API, maybe something like:

class Albums(BaseModel):
    albums: list[Album] = Field(description="list of Albums")

query_str = """\
From every movie that you encounter in the list, generate an example 
album inspired by its title, with an artist and a list of songs.\
"""

index = ... # some big collection of imdb lists, or something
response_synthesizer = get_response_synthesizer(
    # might require some merging of models for cumulative modes 
    response_mode="compact" 
)
query_engine = RetrieverQueryEngine(
    retriever=index.as_retriever(),
    response_synthesizer=response_synthesizer,
)
# would be nice if we could provide the schema at query time, so we 
# can tweak it without rebuilding the whole query engine?
output  = query_engine.query(query_str, output_cls=Albums)

At first glance, ResponseSynthesizer might be the place that needs to be made aware of the output_cls (since the common buck seems to stop at BaseSynthesizer.get_response, which delegates down to LLMPredictor.predict in various ways depending on the response_mode).

But it looks like the query engine code is still using LLMPredictor which doesn't look like it supports functions and function_call like LLM does, so maybe some deeper refactoring to integrate LLM everywhere would be a prerequisite...

[dosubot[bot]]

Hi, @SlapDrone! I'm Dosu, and I'm here to help the LlamaIndex team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you are requesting a feature to integrate function calling APIs with a query engine, similar to the OpenAI Pydantic Program. Logan-markewich suggests creating a PydanticQueryEngine that extracts data for a certain pydantic class. You provided an example of the current single-prompt version and proposed an API that mirrors it. Additionally, you mentioned the need for deeper refactoring to integrate LLM everywhere.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LlamaIndex repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your contribution, and we appreciate your understanding as we work to manage our backlog effectively. If you have any further questions or concerns, please don't hesitate to let us know.

[mphipps2]

This was marked as completed. Was this actually completed? If so, can you point me to the solution?

[logan-markewich]

@mphipps2 yea, you can do this now, see here https://docs.llamaindex.ai/en/stable/optimizing/advanced_retrieval/structured_outputs/query_engine.html