Open chigkim opened 2 months ago
To achieve a comprehensive summary of your entire document split into 74 parts, you should consider using the QASummaryQueryEngineBuilder
from LlamaIndex. This builder allows for the creation of a query engine that is capable of handling both question answering and summarization tasks across multiple document segments. Here's a streamlined approach:
Prepare Your Documents: Make sure each document part is properly represented as a Document
instance.
Instantiate QASummaryQueryEngineBuilder
: This builder is designed to accommodate customization, including the use of a specific Large Language Model (LLM), embedding models, and more.
Build the Query Engine: Utilize the build_from_documents
method to process your documents and create a RouterQueryEngine
. This engine is configured for both QA and summarization tasks, enabling it to generate summaries across all document parts.
Perform Summarization Query: With the RouterQueryEngine
, you can now query for a summary that encompasses all parts of your document.
Regarding the difference between DocumentSummaryIndex
and VectorStoreIndex
, the former is tailored for summarizing documents, ideal for generating quick insights from large texts. In contrast, VectorStoreIndex
is focused on storing and manipulating high-dimensional vector data, which is more aligned with machine learning applications where vector operations are key.
For implementation details and further guidance, you might want to explore the QASummaryQueryEngineBuilder
documentation and examples provided by LlamaIndex. This will give you a clearer understanding of how to set up and use the query engine for document summarization across multiple segments.
To continue the conversation, mention @dosu.
Thanks @dosu. How do you use Ollama as llm for QASummaryQueryEngineBuilder? It wants to use OpenAI instead even though I specified llm parameter.
llm = Ollama(model="llama3", request_timeout=600, base_url=host, additional_kwargs=options)
query_engine_builder = QASummaryQueryEngineBuilder(llm=llm)
ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys
To resolve the issue where specifying Ollama
as the LLM for QASummaryQueryEngineBuilder
results in an error requesting an OpenAI API key, it appears that the system defaults to using OpenAI's LLM, possibly due to how the ServiceContext
is initialized or used within your code. Here's how you can explicitly use your local LLM and avoid this error:
Ollama
in this case) is correctly set in the ServiceContext
. This step is crucial to override any default behavior that attempts to use OpenAI's LLM.from llama_index.llms import LangChainLLM # Adjust this import to match your LLM's specific class
from llama_index import ServiceContext
# Initialize your LLM here, replacing LangChainLLM() with your Ollama initialization
llm = LangChainLLM(model="llama3", request_timeout=600, base_url=host, additional_kwargs=options)
service_context = ServiceContext(llm=llm)
ServiceContext
, set it globally at the start of your application. This approach helps in making sure that all parts of your pipeline are aligned to use the specified LLM.from llama_index import set_global_service_context
set_global_service_context(service_context)
By following these steps, you should be able to bypass the error related to the OpenAI API key and use Ollama
as your LLM for the QASummaryQueryEngineBuilder
. This method ensures that your specified LLM is used throughout your application, avoiding any unintended fallbacks to default settings that might cause issues.
To continue the conversation, mention @dosu.
Question Validation
Question
I'm trying to summarize a long document. I've loaded a pdf document which got splitted into 74 documents by SimpleDirectoryReader. As expected, DocumentSummaryIndex.from_documents goes through each document, and created a summary via the selected llm. However, query_engine.query("Summarize the documents") only selects one node and sends to LLM to summarize the document. How do you make it so that it summarizes the entire document using all nodes? Is there a different QueryEngine or Index I should use for this purpose?
I'm sure there got to be a tutorial on this, but I couldn't find one. The closest one I could find was Document Summary Index - LlamaIndex. However, it's not exactly what I'm looking for. It's more of question and answer system based via embedding/llm. Also, I'm not sure what the advantage of using DocumentSummaryIndex instead of VectorStoreIndex for Q&A.
Thanks so much for your help!