dosubot[bot]
commented
1 year ago 🤖
Hello,
Thank you for bringing this issue to our attention. The error you're encountering is likely due to the fact that AWS Lambda does not support multiprocessing in the same way a traditional operating system does. Each invocation of a Lambda function runs in its own isolated environment, meaning that processes cannot be shared between invocations, and each invocation can only use a single core.
The Pinecone.from_documents function you're using internally calls the Pinecone.add_texts method, which in turn uses the pinecone.Index.upsert method asynchronously. This could be the source of the multiprocessing issue.
To resolve this issue, you could modify the Pinecone.add_texts method to use synchronous requests instead of asynchronous ones. This would involve changing the async_req=True parameter to async_req=False in the self._index.upsert call. Here's how you could modify the method:
def add_texts(
self,
texts: Iterable[str],
metadatas: Optional[List[dict]] = None,
ids: Optional[List[str]] = None,
namespace: Optional[str] = None,
batch_size: int = 32,
embedding_chunk_size: int = 1000,
**kwargs: Any,
) -> List[str]:
# ... existing code ...
for i in range(0, len(texts), embedding_chunk_size):
chunk_texts = texts[i : i + embedding_chunk_size]
chunk_ids = ids[i : i + embedding_chunk_size]
chunk_metadatas = metadatas[i : i + embedding_chunk_size]
embeddings = self._embed_documents(chunk_texts)
for batch in batch_iterate(
batch_size, zip(chunk_ids, embeddings, chunk_metadatas)
):
self._index.upsert(
vectors=batch,
namespace=namespace,
async_req=False, # Changed from True to False
**kwargs,
)
return idsPlease note that this change will make the add_texts method run slower, as it will now wait for each request to complete before sending the next one. However, it should resolve the multiprocessing issue on AWS Lambda.
I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.
Best, Dosu
Sources
- libs/langchain/langchain/vectorstores/pinecone.py
- docs/extras/modules/data_connection/retrievers/how_to/self_query/pinecone.ipynb
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
KLia
LukasFides
bgonzalezfractal
MikeWklm
suresha97
nelsonauner
wcheek
baskaryan
Issue you'd like to raise.
I am writing a few apps that use LangChain and deploying them to AWS Lambda. My previous 2 lambdas are running smoothly, but the one I'm trying to deploy this week is giving me the following stack trace when the lambda is run:
On line 126 in my index_latest_docs.py file, I have the following line:
Pinecone.from_documents(docs, embedding=embeddings, index_name=PINECONE_INDEX_NAME, namespace=PINECONE_NAMESPACE)This is causing it to not run in AWS Lambda because Lambda cannot handle multiprocessing, which the function seems to be doing internally. I don't know if this is a newly introduced issue, because I'm using the same exact function in my other lambda apps, and they seem to be running fine - but maybe the containers were built with a different version than this more recent one.
Any help would be greatly appreciated
Suggestion:
No response