Streaming chat messages can result in large data transfer

[wch]

The transferring of chat messages can result in large amounts of data transferred when the message is streaming. For example, I found that when streaming in a response that was 5800 bytes, it transferred 3.5MB from the Python process to the browser.

(Note that this was counting only the length of the content, and did not include the additional length of the JSON custom message wrapper that is sent to the browser. The wrapper is 187 bytes for each message, and with 1200 individual messages, this results in another 0.2MB.)

This app illustrates. For each chunk, it prints out a line with:

• total number of chunks
• size of content
• cumulative size of all content blocks that have been sent

import ollama

from shiny.express import ui

chat = ui.Chat(id="chat")
chat.ui()

total_count = 0
total_length = 0

@chat.on_user_submit
async def _():
    total_count = 0
    total_length = 0

    messages = chat.messages(format="ollama")
    # Assumes you've run `ollama run llama3.1` to start the server
    response = ollama.chat(
        model="llama3.1",
        messages=messages,
        stream=True,
    )
    await chat.append_message_stream(response)

@chat.transform_assistant_response()
async def transform_response(content: str, chunk: str, done: bool) -> str:
    global total_count, total_length
    total_count += 1
    total_length += len(content)
    print(f"{total_count} : {len(content)} : {total_length}")
    return content

To reduce the amount of data sent, we could do the following:

• Use a text diffing algorithm so that we don't send the entire content over and over as it grows.
• Throttle the responses. I think a ~0.05-0.1s delay still results in a responsive-feeling app. If the throttling results in 2 words sent at a time instead of 1, that would reduce traffic by half.

[cpsievert]

Note to self: Python comes with a difflib module, and ndiff() might be what we need for this https://docs.python.org/3/library/difflib.html#difflib.ndiff