Delayed rendering of user message in WebSocket streaming chatbot example

[pokidyshev]

Description: I'm experiencing a significant delay (around 1 second) between submitting a message and the server's first response when using ws_streaming.py. This delay only occurs for the initial websocket message after hitting submit. Subsequent websocket messages under the same input are processed with low latency. The issue does not occur when using ws.py.

Steps to Reproduce:

1. Open dev tools.
2. Submit a message using ws_streaming.py.
3. Observe a ~1s delay before the server's first response.
4. Verify the delay after the first websocket message in dev tools.
5. Repeat the process with ws.py to verify the absence of the issue.

Expected Behavior: The server should respond without the initial delay when using ws_streaming.py, similar to ws.py.

Observed Behavior: 1-second delay before the first server response when using ws_streaming.py.

Additional Information:

• I've checked the developer tools and confirmed the delay occurs after the first websocket message is sent via HTMX.
• I've attempted to troubleshoot via Google and ChatGPT without success.

Screenshot:

Question: @jph00, any insights or solutions for this issue?

[pokidyshev]

After further debugging and experimentation, I discovered that adding a small asyncio.sleep() delay after clearing the input resolves the WebSocket delay issue:

# Clear the input field for the user
await send(ChatInput())

# This delay unexpectedly fixes the issue
await asyncio.sleep(0.001)

# Stream the model's response
r = cli(messages, sp=sp, stream=True)

I stumbled upon this solution through trial and error and don't fully understand why it works. Any insights or explanations would be greatly appreciated.

[jph00]

How very strange! Have you tried multiple different browsers, and tried disabling any browser extensions you have?

Message ID: @.***>

[pokidyshev]

Same in every browser and without extensions!

As of my understanding, the initial delay when processing the first WebSocket message is caused by the use of a synchronous claudette cli call in an asynchronous context. This synchronous operation blocks the asyncio event loop, preventing it from promptly handling other tasks like WebSocket communication, which led to the noticeable delay.

The asyncio.sleep(0.001) workaround was effective because it forced a context switch, giving the event loop a brief moment to process pending tasks, including WebSocket operations, before the blocking call resumed.

When I switched to LangChain’s asynchronous .astream() method it resolved the issue by allowing the event loop to manage I/O operations and WebSocket tasks concurrently. This change eliminated the need for the sleep hack.

[jph00]

Ah ok great to know - will look into switching over to async for this then.

Message ID: @.***>

[jph00]

All fixed :)