Extremely high CPU usage on the client side during text streaming

oobabooga commented 9 months ago

Describe the bug

During text streaming, the browser uses 100% CPU. This causes an artifact where if the server is streaming fast enough, the frontend will lag behind and will continue streaming for several seconds even if the server gets shut down with Ctrl+C.

This is a severe performance problem with Gradio. See for instance: https://github.com/oobabooga/text-generation-webui/issues/4990

I have tried to find a workaround but had no luck other than limiting updates on the server side to no more than 5 per second.

Have you searched existing issues? 🔎

[X] I have searched and found no existing issues

Reproduction

Run the script below, send a message, and look at the terminal. It will print done but the browser will still be streaming new content.

Look at the CPU utilization in the task manager or htop and it will be at 100%.

import gradio as gr
import random
import time

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.ClearButton([msg, chatbot])

    def respond(message, chat_history):
        bot_message = "How are you?" * 10000
        chat_history.append((message, bot_message))
        for i in range(500):
            chat_history[-1] = (chat_history[-1][0], chat_history[-1][1] + " " + str(i))
            yield "", chat_history

        print("done")

    msg.submit(respond, [msg, chatbot], [msg, chatbot])

demo.queue()
demo.launch(server_name='0.0.0.0')

Screenshot

No response

Logs

No response

System Info

Linux
gradio==4.11.0

It also happens on gradio==3.50.2.

Severity

I can work around it

freddyaboulton commented 8 months ago

@oobabooga can you try with gradio 4.15.0? I think #7055 might have fixed this.

aliabid94 commented 8 months ago

No the core problem of lagging is not fixed actually, working on it now

aliabid94 commented 8 months ago

You should see better performance once #7084 is in, though CPU usage will still be high.

oobabooga commented 7 months ago

CPU usage is still high, but I don't experience any UI lag anymore after recent changes. As that was my main concern, I consider this issue resolved.

gradio-app / gradio