dosubot[bot]
commented
1 year ago Answer generated by a 🤖
Answer
I understand that you're experiencing a memory overflow issue when processing large documents using the Qdrant.from_documents method in your LangChain application. This issue seems to occur before the embeddings interface is called, and you've observed it across different queue tools, suggesting that it's not specific to any particular tool.
The issue appears to be related to the size of the documents you're processing. When the Qdrant.from_documents method tries to process all the documents at once, it can lead to high memory usage and eventually a memory overflow, especially with large documents.
One potential solution to this problem is to process the documents in smaller batches. This can be done by modifying the Qdrant.from_documents method to process a subset of the documents at a time, which should help reduce the memory usage.
Here's how you can modify your code to process the documents in batches:
def handle_embeddings(payload):
loader = UnstructuredPDFLoader(payload["filepath"])
documents = loader.load()
text_splitter = SpacyTextSplitter(pipeline=payload["language"], chunk_size=1536, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings(**payload)
# Process documents in batches
batch_size = 10000 # Adjust this value based on your system's memory capacity
for i in range(0, len(docs), batch_size):
batch_docs = docs[i:i+batch_size]
qdrant = Qdrant.from_documents(
batch_docs, embeddings,
path=path,
collection_name=collection_name,
)In this modified version, the documents are divided into batches of a size that you can specify based on your system's memory capacity, and each batch is processed separately. This approach should significantly reduce the memory usage and help prevent the memory overflow issue.
Please note that this is just one potential solution, and there might be other factors at play. If you're using an older version of LangChain, I would also recommend upgrading to the latest version to ensure you have all the recent optimizations and bug fixes.
If this solution works for you and you believe it's a bug that could impact other users, I encourage you to make a pull request with the proposed changes. Your contribution could help improve the LangChain framework for everyone.
Sources
This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
zhugexinxin
hopkins385
Issue you'd like to raise.
Due to the time-consuming nature of embeddings and storing them in a vector database, I opted for asynchronous queue tasks to handle them. However, I noticed that when processing documents of size 30 MB, the memory usage of the queue task kept increasing until it eventually crashed due to overflow. At this point, I investigated and found that the memory overflow occurred even before the embeddings interface was called, indicating that the issue was with the
Qdrant.from_documentsmethod. I have been searching for the root cause for a while but haven't found it yet.Suggestion:
I hope someone who is familiar with the
Qdrant.from_documentsmethod or has knowledge of other possible causes can help me resolve this issue. The document size of approximately 30 MB corresponds to approximately 560,000 tokens. During the process, I tried using Dramatiq, Celery, and RQ, and encountered the same issue with all of them. Therefore, we can exclude the possibility of the issue being specific to these queue tools.