dosubot[bot]
commented
5 months ago To improve the overall performance of your RAG application, especially when the retriever fails to fetch the document with the exact company name, you can consider the following techniques:
-
Decoupling Chunks Used for Retrieval vs. Chunks Used for Synthesis:
- Embed a document summary, which links to chunks associated with the document: This helps in retrieving relevant documents at a high level before retrieving specific chunks.
- Embed a sentence, which then links to a window around the sentence: This allows for finer-grained retrieval of relevant context.
from llama_index import DocumentSummaryIndex # Create a document summary index doc_summary_index = DocumentSummaryIndex(documents) # Retrieve relevant documents using the summary index relevant_docs = doc_summary_index.retrieve(query) -
Structured Retrieval for Larger Document Sets:
- Metadata Filters + Auto Retrieval: Tag each document with metadata and use these tags to filter documents during retrieval.
- Store Document Hierarchies (summaries -> raw chunks) + Recursive Retrieval: Embed document summaries and map to chunks per document, fetching at the document level first.
from llama_index import MetadataFilter # Tag documents with metadata metadata_filter = MetadataFilter(documents, tags) # Use metadata filter to retrieve documents filtered_docs = metadata_filter.retrieve(query) -
Dynamically Retrieve Chunks Depending on your Task:
- Use task-specific retrieval techniques to handle different types of queries such as summarization or comparisons.
from llama_index import RouterQueryEngine # Initialize the router query engine router_engine = RouterQueryEngine() # Perform task-specific retrieval result = router_engine.query(query, task_type="summarization") -
Optimize Context Embeddings:
- Fine-tune the embedding model to better capture the salient properties of your data corpus.
from llama_index import EmbeddingFineTuner # Fine-tune the embedding model fine_tuner = EmbeddingFineTuner() fine_tuned_model = fine_tuner.fine_tune(corpus) -
Implement Corrective RAG Techniques:
- Evaluate Relevancy: Use a relevancy pipeline to filter out irrelevant documents.
- Transform Queries: Refine the query to improve search performance if initial retrievals are not satisfactory.
from llama_index.core.prompts import PromptTemplate from llama_index.core.query_pipeline.query import QueryPipeline from llama_index.llms.openai import OpenAI DEFAULT_RELEVANCY_PROMPT_TEMPLATE = PromptTemplate( template="""As a grader, your task is to evaluate the relevance of a document retrieved in response to a user's question. Retrieved Document: ------------------- {context_str} User Question: -------------- {query_str} Evaluation Criteria: - Consider whether the document contains keywords or topics related to the user's question. - The evaluation should not be overly stringent; the primary objective is to identify and filter out clearly irrelevant retrievals. Decision: - Assign a binary score to indicate the document's relevance. - Use 'yes' if the document is relevant to the question, or 'no' if it is not. Please provide your binary score ('yes' or 'no') below to indicate the document's relevance to the user question.""" ) DEFAULT_TRANSFORM_QUERY_TEMPLATE = PromptTemplate( template="""Your task is to refine a query to ensure it is highly effective for retrieving relevant search results. \n Analyze the given input to grasp the core semantic intent or meaning. \n Original Query: \n ------- \n {query_str} \n ------- \n Your goal is to rephrase or enhance this query to improve its search performance. Ensure the revised query is concise and directly aligned with the intended search objective. \n Respond with the optimized query only:""" ) class CorrectiveRAGPack(BaseLlamaPack): def __init__(self, documents: List[Document], tavily_ai_apikey: str) -> None: llm = OpenAI(model="gpt-4") self.relevancy_pipeline = QueryPipeline( chain=[DEFAULT_RELEVANCY_PROMPT_TEMPLATE, llm] ) self.transform_query_pipeline = QueryPipeline( chain=[DEFAULT_TRANSFORM_QUERY_TEMPLATE, llm] ) self.llm = llm self.index = VectorStoreIndex.from_documents(documents) self.tavily_tool = TavilyToolSpec(api_key=tavily_ai_apikey) def retrieve_nodes(self, query_str: str, **kwargs: Any) -> List[NodeWithScore]: retriever = self.index.as_retriever(**kwargs) return retriever.retrieve(query_str) def evaluate_relevancy(self, retrieved_nodes: List[Document], query_str: str) -> List[str]: relevancy_results = [] for node in retrieved_nodes: relevancy = self.relevancy_pipeline.run( context_str=node.text, query_str=query_str ) relevancy_results.append(relevancy.message.content.lower().strip()) return relevancy_results def extract_relevant_texts(self, retrieved_nodes: List[NodeWithScore], relevancy_results: List[str]) -> str: relevant_texts = [ retrieved_nodes[i].text for i, result in enumerate(relevancy_results) if result == "yes" ] return "\n".join(relevant_texts) def search_with_transformed_query(self, query_str: str) -> str: search_results = self.tavily_tool.search(query_str, max_results=5) return "\n".join([result.text for result in search_results]) def get_result(self, relevant_text: str, search_text: str, query_str: str) -> Any: documents = [Document(text=relevant_text + "\n" + search_text)] index = SummaryIndex.from_documents(documents) query_engine = index.as_query_engine() return query_engine.query(query_str) def run(self, query_str: str, **kwargs: Any) -> Any: retrieved_nodes = self.retrieve_nodes(query_str, **kwargs) relevancy_results = self.evaluate_relevancy(retrieved_nodes, query_str) relevant_text = self.extract_relevant_texts(retrieved_nodes, relevancy_results) search_text = "" if "no" in relevancy_results: transformed_query_str = self.transform_query_pipeline.run( query_str=query_str ).message.content search_text = self.search_with_transformed_query(transformed_query_str) if search_text: return self.get_result(relevant_text, search_text, query_str) else: return self.get_result(relevant_text, "", query_str) -
Utilize RAGatouille Retriever Pack:
- RAGatouille Retriever Pack: This pack uses the RAGPretrainedModel for indexing and retrieval, which can be more effective in handling complex queries and large document sets.
from llama_index.packs.ragatouille_retriever.base import RAGatouilleRetrieverPack # Initialize the RAGatouille Retriever Pack retriever_pack = RAGatouilleRetrieverPack(documents) # Add documents to the retriever retriever_pack.add_documents(new_documents) # Run the retriever with a query result = retriever_pack.run(query)
By implementing these techniques, you can enhance the retrieval accuracy and overall performance of your RAG application [1][2][3].
To continue talking to Dosu, mention @dosu.
avalonliberty
Question Validation
Question
Hi, All. I loaded an stored the table into chromadb. The table contains four columns ["company_name", "country", "city", "description"]. After that, I query the llm with the query "Do you know the company XXX ?". Though, the retriever failed to fetch the document with exact the same company name. I did use the reranking service to improve the quality but it did not work given that the retriever had failed to retrieve relevant documents in the returned documents. I want to know how I can improve the overall performance of the RAG application.
Here is the code I used to do the whole thing. Some of them are abstract. Please let me know if you need more detail to solve this this issue.