Open Tolkoton opened 1 month ago
Hi @Tolkoton, there's not enough context here. Whether it's synchronous or not depends on the implementation of get_rag_chain
. What does get_rag_chain
do?
If this is from existing LangChain code, would you be kind enough to add a link to?
async def get_rag_chain(embeddings, collection_name):
try:
vector_store = ZillizDB.create_vector_store(embeddings=embeddings, collection_name=collection_name)
print(f"Zilliz vector store created for collection: {collection_name}")
retriever = vector_store.as_retriever()
print("Retriever initialized.")
template = """Дай обґрунтовану відповідь на питання користуючись контекстом."
{context}
Question: {question}
Helpful Answer:"""
model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-v2-m3")
compressor = CrossEncoderReranker(model=model, top_n=15)
compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)
rag_prompt = PromptTemplate.from_template(template)
rag_chain = (
{"context": compression_retriever, "question": RunnablePassthrough()}
| rag_prompt
| LLM)
print("RAG chain set up.")
return rag_chain, compression_retriever
except Exception as e:
print(f"Error setting up RAG chain: {e}")
return None, None
@Tolkoton , I see in your code above that you use LLM in your rag_chain, what specific LLM are you using, the behaviour you are seeing could be related to that specific LLM.
In particular if no _acall is implemented for the LLM a "naive run_in_executor" approach is used, in that scenario a blocking instruction in the _call method could be the cause of what you are seeing.
For further investigations I think that the LLM used is crucial.
If no _acall implementation is present this would be a feature request.
@eyurtsev please correct me if I'm wrong.
@pm390 @eyurtsev I am using gpt4 turbo. But the most time is consumed in Milvus search and reranking, and its most likely problem there.
What does this code do? Could you link to it? I can't find it in the code base, but it's also clearly not async. I'm guessing it makes network calls.
vector_store = ZillizDB.create_vector_store(embeddings=embeddings, collection_name=collection_name)
@eyurtsev @staticmethod def create_vector_store(embeddings, collection_name, index_params=INDEX_PARAMS, search_params=SEARCH_PARAMS): vector_store = Milvus( embedding_function=embeddings, connection_args={'uri': ZILLIZ_CLUSTER_ENDPOINT, 'token': ZILLIZ_TOKEN}, collection_name=collection_name, search_params=search_params, index_params=index_params) return vector_store
@Tolkoton If that Milvus class you are using is this one, I can see that it instantiates Collection Object from pymilvus internally.
I tried digging into that object and saw that internally is initialized and it uses using grpc_handler (also this from pymilvus). Checking inside grpc_handler's methods I see that they perform calls using grcp stub and get a future from them, but in almost all methods they proceed to call .result() on that future, blocking execution there until a response is returned.
I'm not quite sure if this is the only reason, hope this helps further investigation.
I think also the problem could be present for later searching on the vector store, currently no custom implementation of the asynchronous methods is provided so the implementation of the base class is used (so just invoking the syncrhonous method using run_in_executor).
Some methods like search of pymilvus collections can be invoked asynchronously by passing _async=True as a kwarg to return a concurrent.futures.Future object instead of making blocking calls.
Milvus is not supporting asynchronous calls.
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
No response
Description
This code is working, but its not asyncronous. the single point that takes a lot of time in all the executions is this: response = await get_response(collection_name=collection_name, user_input=user_input) it blocks system for all other users, so the ainvoke must be not working as expected
System Info
System Information
Package Information
Packages not installed (Not Necessarily a Problem)
The following packages were not found: