dosubot[bot]
commented
1 year ago Answer generated by a 🤖
Answer
Hello,
Thank you for bringing this issue to our attention. The problem you're facing is due to duplicate ids causing a mismatch between the faiss index and the document's index. This is because the __add function in FAISS class doesn't handle the case where the ids provided by the user are not unique.
To resolve this issue, you can add a check to ensure that the ids are unique before adding them to the index_to_docstore_id dictionary. Here's how you can modify the __add function to handle this:
def __add__(
self,
texts: Iterable[str],
embeddings: Iterable[List[float]],
metadatas: Optional[List[dict]] = None,
ids: Optional[List[str]] = None,
**kwargs: Any,
) -> List[str]:
if not isinstance(self.docstore, AddableMixin):
raise ValueError(
"If trying to add texts, the underlying docstore should support "
f"adding items, which {self.docstore} does not"
)
documents = []
for i, text in enumerate(texts):
metadata = metadatas[i] if metadatas else {}
documents.append(Document(page_content=text, metadata=metadata))
if ids is None:
ids = [str(uuid.uuid4()) for _ in texts]
else:
# Check if ids are unique
if len(ids) != len(set(ids)):
raise ValueError("Ids should be unique")
# Add to the index, the index_to_id mapping, and the docstore.
starting_len = len(self.index_to_docstore_id)
faiss = dependable_faiss_import()
vector = np.array(embeddings, dtype=np.float32)
if self._normalize_L2:
faiss.normalize_L2(vector)
self.index.add(vector)
# Get list of index, id, and docs.
full_info = [(starting_len + i, ids[i], doc) for i, doc in enumerate(documents)]
# Add information to docstore and index.
self.docstore.add({_id: doc for _, _id, doc in full_info})
index_to_id = {index: _id for index, _id, _ in full_info}
self.index_to_docstore_id.update(index_to_id)
return [_id for _, _id, _ in full_info]This modification will raise a ValueError if the ids provided by the user are not unique, thus preventing the mismatch issue.
Please let me know if this resolves your issue or if you need further assistance.
Best, Dosu
Sources
This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
xia-huang-411303
System Info
windows 10 python 3.10.12
Who can help?
@eyurtsev There is a BUG in the def __add function in langchain\vectorstores\faiss.py: When the parameter ids is provided and there are duplicates in it, index_to_docstore_id will report an error and the program will terminate. But the text vector embedding has been added to the index. This causes the id in the index to be inconsistent with the id in index_to_docstore_id, not aligned, and the text cannot be matched correctly.
langchain\vectorstores\faiss.py 中 def __add 函数存在BUG: 当参数 ids 已提供, 且其中存在重复时, index_to_docstore_id 新增会报错,程序运行终止. 但是文本向量 embedding 已经添加至 index 中.导致 index 中的 id 与 index_to_docstore_id 的中 id 对应不一致,文本无法正确匹配
Information
Related Components
Reproduction
None
Expected behavior
None