run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
35.18k stars 4.94k forks source link

Can RecursiveRetriever be used with QueryFusionRetriever? #14713

Closed hingkan closed 1 month ago

hingkan commented 1 month ago

Question Validation

Question

DATA: Split the document data into multiple "parent" chunks, and then further subdivide each block into "child" chunks.

I originally wanted to first build a hybrid recursive search using the RecursiveRetriever retriever and the QueryFusionRetriever retriever together. Enter RecursiveRetriever in QueryFusionRetriever.

CODE: ` from llama_index.core.retrievers import QueryFusionRetriever

retriever = QueryFusionRetriever( [recursive_retriever1, recursive_retriever2, ...], similarity_top_k=2, num_queries=4, # set this to 1 to disable query generation use_async=True, verbose=True,

query_gen_prompt="...", # we could override the query generation prompt here

) `

ERROR: ValueError: Query id 23ca7db4-1f10-4236-88c9-b0102081b8b1 not found in either retriever_dict or query_engine_dict.

logan-markewich commented 1 month ago

It can, but it seems like something is wrong with one of your recursive retrievers

I bet you'd get a similar error with recursive_retriever1.retrieve("query")

hingkan commented 1 month ago

It can, but it seems like something is wrong with one of your recursive retrievers

I bet you'd get a similar error with recursive_retriever1.retrieve("query")

Sure, and indeed it has already gone wrong in RecursiveRetriever. I found the problem, RecursiveRetriever must pass the "node_dict" argument. Thank you.

hingkan commented 1 month ago

RecursiveRetriever in return only the retriever, how to use together with PrevNextNodePostprocessor, for how can achieve the same function as PrevNextNodePostprocessor.

Prerequisite: RecursiveRetriever is used together with QueryFusionRetriever.

logan-markewich commented 1 month ago

Postprocessors are generally used after retrieval. So you run your query fusion retriever. Then run your postprocessor on the results

If you want to postprocess individual retrievers inside the query fusion retriever, i would write a custom retriever

hingkan commented 1 month ago

Postprocessors are generally used after retrieval. So you run your query fusion retriever. Then run your postprocessor on the results

If you want to postprocess individual retrievers inside the query fusion retriever, i would write a custom retriever

How does CustomRetriever implement a single retriever postprocess in QueryFusionRetriever.