dosubot[bot]
commented
7 months ago The "division by zero" error you're encountering with the BM25Retriever typically happens when the algorithm tries to calculate scores for a query against an empty corpus, or the documents in your corpus result in no tokens after preprocessing. This can occur if the documents are empty, or if the tokenization process filters out all content from the documents.
To resolve this, ensure your corpus is not empty and contains valid, tokenizable content. If your corpus is valid but you're still facing this issue, it might be due to the tokenization process removing all tokens from your documents, leaving them empty. In such cases, you might need to adjust the tokenization process to ensure that documents are left with tokens after preprocessing.
Here's a modification you can make to the BM25Retriever initialization to filter out empty token lists and avoid the "division by zero" error:
class BM25Retriever(BaseRetriever):
def __init__(
self,
nodes: List[BaseNode],
tokenizer: Optional[Callable[[str], List[str]]] = None,
similarity_top_k: int = 100,
verbose: bool = False,
) -> None:
self._nodes = nodes
self._tokenizer = tokenizer or default_tokenizer
# Tokenize the content of each node and filter out empty token lists
self._corpus = [self._tokenizer(node.content) for node in self._nodes if self._tokenizer(node.content)]
if not self._corpus:
raise ValueError("The corpus is empty or all documents resulted in no tokens after tokenization.")
self.bm25 = BM25Okapi(self._corpus)This ensures that the corpus passed to BM25Okapi does not contain any empty token lists, which should help in avoiding the "division by zero" error. Additionally, it raises a ValueError if the corpus is empty or if all documents result in no tokens after tokenization, allowing for easier debugging and error handling.
Please try adjusting your tokenization process or ensuring your corpus contains valid content and let me know if the issue persists.
Sources
- [Bug]: BM25 retriever not working just with indexes
- llama-index-integrations/retrievers/llama-index-retrievers-bm25/llama_index/retrievers/bm25/base.py
- llama-index-legacy/llama_index/legacy/retrievers/bm25_retriever.py
logan-markewich
mw19930312
abdelatifsd
Bug Description
I'm following the tutorial https://docs.llamaindex.ai/en/stable/examples/retrievers/reciprocal_rerank_fusion/ to create a fusion retriever. However, I encounter an error of division by zero in bm25 retriever.
I've provided a code example below. Everything works before I added the line of
retriever=retrieverin CitationQueryEngine.Can anyone take a look and provide some help? Or at least let me know how I should debug? Thanks!
Version
0.10.16
Steps to Reproduce
from llama_index.retrievers.bm25 import BM25Retriever from llama_index.core.retrievers import QueryFusionRetriever vector_retriever = index.as_retriever(similarity_top_k=config.top_k)
bm25_retriever = BM25Retriever.from_defaults( docstore=index.docstore, similarity_top_k=config.top_k )
retriever = QueryFusionRetriever( [vector_retriever, bm25_retriever], similarity_top_k=config.top_k, num_queries=4, # set this to 1 to disable query generation mode="reciprocal_rerank", use_async=True, verbose=True, ) citation_query_engine = CitationQueryEngine.from_args( llm=final_response_llm, retriever=retriever, index=index,
here we can control how granular citation sources are, set to match the max ingestion chunk size
)
Relevant Logs/Tracbacks