dosubot[bot]
commented
8 months ago 🤖
Hello @daje0601! 🚀
I'm Dosu, a friendly bot here to lend a hand while we wait for a human maintainer to arrive. As a bot, I can help you troubleshoot bugs, answer your queries, and even guide you on how to contribute to this project. Let's make your LangChain experience as smooth as possible together! 💻
Don't worry, we'll get to the bottom of your issue in no time!
The error you're encountering is due to the tokenizer used in the Embeddings object not having a padding token set. This is necessary for the Embeddings object to correctly process the input texts.
To resolve this issue, you need to ensure that the tokenizer used in your Embeddings object has a padding token. If you're using a tokenizer from the Hugging Face's Transformers library, you can set the padding token like this:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer.pad_token = tokenizer.eos_tokenThen, you can use this tokenizer in your Embeddings object:
from langchain_community.embeddings.openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(tokenizer=tokenizer)Now, you should be able to use the FAISS.from_documents function without encountering the ValueError.
The FAISS.from_documents function in the LangChain framework is a class method that constructs a FAISS (Facebook AI Similarity Search) wrapper from raw documents. It takes an iterable of tuples, where each tuple contains a text and its corresponding embedding vector, and an Embeddings object. It also optionally takes metadata and ids for the documents.
The function is used to initialize a FAISS object with pre-computed embeddings of the documents. This is useful when you want to perform similarity searches on the documents using the FAISS library, which is designed for efficient similarity search and clustering of dense vectors.
Here is the relevant code:
@classmethod
def from_embeddings(
cls,
text_embeddings: Iterable[Tuple[str, List[float]]],
embedding: Embeddings,
metadatas: Optional[Iterable[dict]] = None,
ids: Optional[List[str]] = None,
**kwargs: Any,
) -> FAISS:
"""Construct FAISS wrapper from raw documents."""
texts = [t[0] for t in text_embeddings]
embeddings = [t[1] for t in text_embeddings]
return cls.__from(
texts,
embeddings,
embedding,
metadatas=metadatas,
ids=ids,
**kwargs,
)In the example provided in the code, the function is used as follows:
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
text_embeddings = embeddings.embed_documents(texts)
text_embedding_pairs = zip(texts, text_embeddings)
faiss = FAISS.from_embeddings(text_embedding_pairs, embeddings)In this example, the OpenAIEmbeddings object is used to compute the embeddings of the texts, and then the from_embeddings function is used to create a FAISS object from these embeddings.
I hope this helps! If you have any other questions, feel free to ask.
Best, Dosu
Sources
- libs/community/langchain_community/vectorstores/faiss.py
- libs/langchain/langchain/vectorstores/faiss.py
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
daje0601
TimmekHW
Checked other resources
Example Code
I use this code. I met this error. Why occur this occur? I look around all. I can't fix ti. so Sad ㅠㅠㅠ Could you explain me how to get fix it?
Description
I use this code. I met this error. Why occur this occur? I look around all. I can't fix ti. so Sad ㅠㅠㅠ Could you explain me how to get fix it?
System Info
langchain 0.1.0 langchain-community 0.0.12 langchain-core 0.1.10 langsmith 0.0.80
Related Components