Me and some colleagues observed an issue, where Langchain Callback Handlers get not triggered if passed as callbacks on Clientside within the Streaming function. The Runnable contains RAG Components like a Retriever. The chain/runnable was tested without Langserve and the callbacks were triggered (on_retriever_end, on_new_llm_token, on_chat_model_start) just as expected. Using Langserve's RemoteRunnable, this behaviour changed.
Are we missing out on some important information regarding clientside streaming on a RemoteRunnable or could this be an implementation Issue?
The following code shows, how it looks like
def azure_ai_chat():
return RemoteRunnable("http://localhost:8001/azure-ai-search/")
# Langserve Runnable: Testes and Works as endpoint as well as without Langserve
runnable = azure_ai_chat() # type: Runnable
ai_response = cl.Message(content="", elements=[], actions=[])
token_usage = TokenUsage()
async for chunk in runnable.astream(
{"question": message.content},
config=RunnableConfig(callbacks=[cl.LangchainCallbackHandler(),
TokenUsageCallbackHandler(
"gpt-4", token_usage),
DocsMessageHandler(ai_response)
]),
):
# Stream specific code
if "answer" in chunk.keys():
await response_msg.stream_token(chunk["answer"].content)
if chunk and "context" in chunk.keys():
relevant_docs = chunk["context"]
Me and some colleagues observed an issue, where Langchain Callback Handlers get not triggered if passed as callbacks on Clientside within the Streaming function. The Runnable contains RAG Components like a Retriever. The chain/runnable was tested without Langserve and the callbacks were triggered (on_retriever_end, on_new_llm_token, on_chat_model_start) just as expected. Using Langserve's RemoteRunnable, this behaviour changed.
Are we missing out on some important information regarding clientside streaming on a RemoteRunnable or could this be an implementation Issue?
The following code shows, how it looks like