Closed gbaian10 closed 3 months ago
Hi there! tl;dr
async def call_model(state: State, config: RunnableConfig):
messages = state["messages"]
response = await model.ainvoke(messages, config)
instead of
async def call_model(state: State:
messages = state["messages"]
response = await model.ainvoke(messages)
Why? Python asyncio pre-3.11 doesn't support context propagation through tasks, so the streaming tracer that is used to stream out all the traced steps isn't seen by the model unless you explicitly pass the configuration from the node to nested runs. I'll update the doc to emphasize this.
LMK if this doesn't fix the issue for you and we can re-open this issue!
Hi there! tl;dr
async def call_model(state: State, config: RunnableConfig): messages = state["messages"] response = await model.ainvoke(messages, config)
instead of
async def call_model(state: State: messages = state["messages"] response = await model.ainvoke(messages)
Why? Python asyncio pre-3.11 doesn't support context propagation through tasks, so the streaming tracer that is used to stream out all the traced steps isn't seen by the model unless you explicitly pass the configuration from the node to nested runs. I'll update the doc to emphasize this.
LMK if this doesn't fix the issue for you and we can re-open this issue!
Thank you, I succeeded.
By the way, the async nodes
hyperlink at the top of the same page is currently invalid.(404)
Issue with current documentation:
https://langchain-ai.github.io/langgraph/how-tos/streaming-tokens/#streaming-llm-tokens
Sample code does not produce expected results. ("on_chat_model_stream" part)
This is my code
output:
python = 3.10 langgraph = 0.0.62 langchain-core = 0.2.3 langchain-openai = 0.1.8
Idea or request for content:
No response