Streaming complex object fast in custom component makes bug

MobisParkHeekang commented 4 months ago

Describe the bug

I define a complex object like below:

class TextSection(GradioModel):
    section_type: Literal["text"]
    section_name: Optional[str]
    section_state: Optional[Literal["initial", "running", "complete"]]
    section_id: str
    feedback: Optional[bool]
    section_content: str

class Bubble(GradioModel):
    role: Literal["assistant", "user", "system"]
    sections: List[TextSection]

class ChatbotData(GradioModel):
    bubbles: List[Bubble]
    status: Literal["initial", "running", "complete"]

class GradioData(GradioRootModel):
    root: ChatbotData

When I stream data with yield, errors emerge.

def run(query, chat):
    chat = {
        "bubbles": [
            {
                "role": "user",
                "sections": [
                    {
                        "section_type": "text",
                        "section_name": null,
                        "section_state": "complete",
                        "section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
                        "feedback": null,
                        "section_content": ""
                    }
                ]
            },
            (assistant_bubble := {
                "role": "assistant",
                "sections": [
                    {
                        "section_type": "text",
                        "section_name": null,
                        "section_state": "running",
                        "section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
                        "feedback": null,
                        "section_content": ""
                    }
                ]
            })
        ],
        "status": "running"
    }
    yield chat

    text = "Lorem ipsum dolor sit amet"
    for c in text:
        assistant_bubble["sections"][0]["section_content"] += c
        yield chat

    #time.sleep(0.1)
    assistant_bubble["sections"][0]["section_status"] = "complete"
    chat["status"] = "complete"
    yield chat

SSE data suddenly changes from below:

{
    "msg": "process_generating",
    "event_id": "2c9694b94b054f76a254d05e143e7f6f",
    "output": {
        "data": [
            {
                "bubbles": [
                    {
                        "role": "user",
                        "sections": [
                            {
                                "section_type": "text",
                                "section_name": null,
                                "section_state": "complete",
                                "section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
                                "feedback": null,
                                "section_content": ""
                            }
                        ]
                    },
                    {
                        "role": "assistant",
                        "sections": [
                            {
                                "section_type": "text",
                                "section_name": null,
                                "section_state": "running",
                                "section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
                                "feedback": null,
                                "section_content": ""
                            }
                        ]
                    }
                ],
                "status": "running"
            }
        ],
        "is_generating": true,
        "duration": 0.0014843940734863281,
        "average_duration": 0.0003377993901570638
    },
    "success": true
}

to below:

{
    "msg": "process_generating",
    "event_id": "2c9694b94b054f76a254d05e143e7f6f",
    "output": {
        "data": [
            [
                [
                    "replace",
                    [
                        "bubbles",
                        0,
                        "sections",
                        0,
                        "section_id"
                    ],
                    "b1155e6f-1349-11ef-ba0d-8f639724379a"
                ],
                [
                    "replace",
                    [
                        "bubbles",
                        1,
                        "sections",
                        0,
                        "section_id"
                    ],
                    "b11561a3-1349-11ef-a2a4-8f639724379a"
                ],
                [
                    "append",
                    [
                        "bubbles",
                        1,
                        "sections",
                        0,
                        "section_content"
                    ],
                    "L"
                ]
            ]
        ],
        "is_generating": true,
        "duration": 0.0003814697265625,
        "average_duration": 0.0003392081106862714
    },
    "success": true
}

However, if I put time.sleep()(currently commented out), errors do not emerge.

Have you searched existing issues? 🔎

[X] I have searched and found no existing issues

Reproduction

types

class TextSection(GradioModel):
    section_state: Optional[Literal["initial", "running", "complete"]]
    section_content: str

class Bubble(GradioModel):
    role: Literal["assistant", "user", "system"]
    sections: List[TextSection]

class ChatbotData(GradioModel):
    bubbles: List[Bubble]
    status: Literal["initial", "running", "complete"]

class GradioData(GradioRootModel):
    root: ChatbotData

streaming function

def run(query, chat):
    chat = {
        "bubbles": [
            {
                "role": "user",
                "sections": [
                    {
                        "section_type": "text",
                        "section_content": query
                    }
                ]
            },
            (assistant_bubble := {
                "role": "assistant",
                "sections": [
                    {
                        "section_type": "text",
                        "section_content": ""
                    }
                ]
            })
        ],
        "status": "running"
    }
    yield chat

    text = "Lorem ipsum dolor sit amet"
    for c in text:
        assistant_bubble["sections"][0]["section_content"] += c
        yield chat

    #time.sleep(0.1)
    assistant_bubble["sections"][0]["section_status"] = "complete"
    chat["status"] = "complete"
    yield chat

Screenshot

No response

Logs

No response

System Info

Gradio Environment Information:
------------------------------
Operating System: Linux
gradio version: 4.25.0
gradio_client version: 0.15.0

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
altair: 5.2.0
fastapi: 0.110.0
ffmpy: 0.3.2
gradio-client==0.15.0 is not installed.
httpx: 0.27.0
huggingface-hub: 0.20.3
importlib-resources: 6.1.2
jinja2: 3.1.2
markupsafe: 2.1.3
matplotlib: 3.8.3
numpy: 1.26.4
orjson: 3.9.15
packaging: 23.1
pandas: 2.2.0
pillow: 9.5.0
pydantic: 2.6.1
pydub: 0.25.1
python-multipart: 0.0.9
pyyaml: 6.0.1
ruff: 0.2.2
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.9.0
typing-extensions: 4.8.0
uvicorn: 0.27.1
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.

gradio_client dependencies in your environment:

fsspec: 2023.10.0
httpx: 0.27.0
huggingface-hub: 0.20.3
packaging: 23.1
typing-extensions: 4.8.0
websockets: 11.0.3

Severity

Blocking usage of gradio

freddyaboulton commented 4 months ago

Hi @MobisParkHeekang ! What exactly is the error that's happening? It's expected that the data format sent from the server to change because we will only send the diff. Can you please share the custom component you're developing?

MobisParkHeekang commented 4 months ago

Thank you for your reply. Unfortunately, the component is on the company's private server, so it cannot be shared. However, your expectation sounds really plausible. Can you please tell me in which file the diff calculation and sending occur?

abidlabs commented 4 months ago

Hi @MobisParkHeekang you may want to check this out:

https://github.com/gradio-app/gradio/blob/8f46556b38e35cffbadac74ff80445dceea3bcf5/gradio/blocks.py#L1792

MobisParkHeekang commented 4 months ago

Can you please tell me in which js(or svelte) file ‘rebuild’ value from diffs sent from FastAPI server?

freddyaboulton commented 4 months ago

The js client will rebuild the value based on the diff: https://github.com/gradio-app/gradio/blob/35905c5c8f7acbe669486ac8f57b6955328e4783/client/js/src/utils/submit.ts#L597

gradio-app / gradio