[Question]: How to pass chat_history to Router Component

[Hassan-Haq]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I am working on building an orchestrator for a chat system using llamaIndex Query pipelines. I need a little help with a special case where I need the Router to select an Agent based on items mentioned in the users query.

In the pipeline the query is initially passed to a function to see if there's multiple versions of an item in the db, if there's multiple versions of the item we pull those ids and append them to chat_history

Following this at the routing step if there's multiple versions present I want the Router to select a certain agent that will prompt the user to clarify their query by picking one of the items. To do this, when we set-up the LLM single selector I intend to update it's prompt to contain the context_str as well as the modified chat_history.

The problem I am running into is that the Router has access to the query and a list of agents but I don't know how I can access the chat_history at this level. It needs access to this object to properly select the correct Agent. Is there a way to override this to pass in the history?

Below is how I've set-up the pipeline.

def orchestrator(
    *,
    is_verbose: bool = False,
) -> Orchestrator:
    event_queue: Queue[OrchestratorEvent] = Queue()
    callback_manager = get_callback_manager(event_queue=event_queue)

    # Agents
    agents: list[OrchestratorAgent] = [...list of agents]

    if len(agents) == 0:
        raise ValueError("No agent type specified")

    selector = LLMSingleSelector.from_defaults(llm=instance_configs.llm)
    router = RouterComponent(
        selector=selector,
        choices=list(map(lambda n: n.description_prompt, agents)),
        components=typing.cast(list[QUERY_COMPONENT_TYPE], agents),
        verbose=is_verbose,
    )
    query_rewriter = QueryRewriter(
        llm=instance_configs.query_rewriter_llm, callback_manager=callback_manager
    )

    chain = [get_versions, query_rewriter, router]

    query_pipeline = QueryPipeline(
        chain=chain,
        callback_manager=callback_manager,
        verbose=is_verbose,
    )
    return Orchestrator(queue=event_queue, agents=agents, query_pipeline=query_pipeline)

I am trying currently to override the RouterComponent but I am not sure how to go about this. 

[logan-markewich]

The chat history can just be part of the query as input to the router

Just need a component to combine the rewritten query with chat history it looks like

[logan-markewich]

(FYI, query pipelines are a little unmaintained, in favor of workflows, which are much more straightforward to work with)

https://docs.llamaindex.ai/en/stable/module_guides/workflow/#workflows

[Hassan-Haq]

Sorry I don't think I quite understand. Chat history isn't available at this level. How would i combine chat_history w/ the query? coming out of the get_versions method? the query_rewriter here is optional I just left that condition out of the code.

[Hassan-Haq]

(FYI, query pipelines are a little unmaintained, in favor of workflows, which are much more straightforward to work with)

https://docs.llamaindex.ai/en/stable/module_guides/workflow/#workflows

Will be picking up workflows soon as well!

[AbhishekRP2002]

@Hassan-Haq you can leverage state storage at the Control plane of your agent and Message Queue in conjunction with workflows to handle this issue I believe. https://www.llamaindex.ai/blog/introducing-llama-deploy-a-microservice-based-way-to-deploy-llamaindex-workflows maybe u have already gone through this one, but still sharing if incase it helps you refresh !

[Hassan-Haq]

I looked into setting up storage state but it seems like overkill just to access chat history in this case. I was hoping there as an elegant solution to access data in the pipeline