`ValueError: ('Event not found')`

KeremTurgutlu commented 1 year ago

Describe the bug

I've a gradio web UI which I've deployed in a google cloud VM. Everything works fine expect when I try to use the demo from my custom domain url mydomain.com it fails and hangs forever once an event is triggered.

When I trigger an event UI just hangs indefinitely, from the looks there seems to be a new event id which can't be found and it keeps trying with a new one. This just keeps on going with a new event id everytime and doesn't stop unless the browser is closed.

In Google Cloud I've opened HTTP, HTTPS and a custom TCP Port 7860.

After launching the demo I can successfully use the demo if I access it from https://External-ID:7860 however when I try to use it from my custom domain URL, it fails whenever I trigger an action such as clicking the submit button, changing slider values etc..This can also be seen in the logs.

Have you searched existing issues? 🔎

[X] I have searched and found no existing issues

Reproduction

Demo UI

import gradio as gr
from gradio.components import Textbox, Radio
import json
from datetime import datetime
import pandas as pd
import os

# TODO: Add functionality to jump to any task by index, 
# also show if a task is already rated or not.
# Show a green checkmark if rated, red cross if not rated using Gradio.
# Workers should be able to overwrite their ratings if they want to.
class RateTexts:
    def __init__(self, ratings_filename=None, resume_from_filename=None, debug=False):

        if resume_from_filename is None:
            self.ratings = {}
        else:
            with open(resume_from_filename, "r") as f:
                self.ratings = json.load(f)

        self.current_index = 0

        # Sample list of tuples
        self.texts = [
            ("Main Text 1", "Option A1", "Option B1", "Option C1"),
            ("Main Text 2", "Option A2", "Option B2", "Option C2"),
            ("Main Text 3", "Option A3", "Option B3", "Option C3"),
            ("Main Text 4", "Option A4", "Option B4", "Option C4"),
        ]

        # load texts from file if None raise erro
        if not debug:
            if ratings_filename is None:
                raise ValueError("Please provide a ratings csv filename with 4 columns, 1 main text and 3 options text.")

            if os.path.exists(ratings_filename):    
                self.texts = [tuple(o) for o in pd.read_csv(ratings_filename).values]
            else:
                raise ValueError(f"Ratings filename {ratings_filename} doesn't exist.")

    def rate_texts(self, rating):
        # Save the rating
        if self.current_index < 0:
            self.current_index += 1
        elif rating is None:
            gr.Warning("Please select a rating before Submit.")
            return *self.texts[self.current_index], self.create_html_status()
        else:
            self.ratings[str(self.current_index)] = {"index":self.current_index, "rating":rating}
            self.current_index += 1

        if self.current_index < len(self.texts):
            return *self.texts[self.current_index], self.create_html_status()

        return (f"You've rated all texts! Don't forget to click 'Save' and click 'Reset' if you want to start over.", "", "", "", self.create_html_status())

    def reset(self):
        self.ratings = {}
        self.current_index = 0
        return self.rate_texts(None)

    def save_ratings(self):
        # save with timestamp
        filename = f"saved_ratings/ratings_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
        with open(filename, "w") as f:
            print(self.ratings)
            json.dump(self.ratings, f)

    def show_ratings(self):
        return self.ratings

    def create_html_status(self):
        # write html text which display green checkmark if rated, red cross if not rated.
        example_rating_status = f"At Example {self.current_index} is Rated: " + ("✅" if str(self.current_index) in self.ratings else "❌")
        return gr.HTML(value=f"<h3>Number of texts rated so far: {len(self.ratings)}/{len(self.texts)}</h3>"
                             f"<h4>{example_rating_status}</h4>")

def reset_radio():
    return gr.Radio(label="Your Rating", 
                    choices=["Option A", "Option B", "Option C", "All similar"], 
                    value=None)

def create_demo(rater):
    with gr.Blocks(gr.themes.GoogleFont) as demo:

        gr.Markdown(f"Click 'Submit' to begin. Once you see the 'Input LLM Text' and 'Option A', 'Option B', 'Option C' texts, you can click 'Submit' after selecting 'Your Rating'.\n\n"
                    "At any given time you can click 'Show Ratings' to see the ratings you've given so far.\n\n")

        # A slider to move between texts by index
        def set_rater_index(slider_value):
            rater.current_index = slider_value
            return *rater.texts[rater.current_index], rater.create_html_status()

        # write html to show the number of rated texts so far.
        completion_status_html = rater.create_html_status()

        slider = gr.Slider(label="Text Index", minimum=0, maximum=len(rater.texts)-1, step=1, value=rater.current_index)        
        main_text = Textbox(label="Input LLM Text", min_width=1000, lines=10, value=rater.texts[rater.current_index][0])
        # add side by side textboxes
        with gr.Row():
            lines = 20
            option1 = Textbox(label=f"Option A", text_align="left", lines=lines, value=rater.texts[rater.current_index][1])
            option2 = Textbox(label=f"Option B", text_align="left", lines=lines, value=rater.texts[rater.current_index][2])
            option3 = Textbox(label=f"Option C", text_align="left", lines=lines, value=rater.texts[rater.current_index][3])

        slider.release(set_rater_index, inputs=[slider], outputs=[main_text, option1, option2, option3, completion_status_html])

        with gr.Row():
            with gr.Column():
                radio = gr.Radio(label="Your Rating", choices=["Option A", "Option B", "Option C", "All similar"])
            with gr.Column():
                submit_btn = gr.Button(value="Submit")
                reset_button = gr.Button(value="Reset")
                save_btn = gr.Button(value="Save")

                submit_btn.click(rater.rate_texts, inputs=[radio], outputs=[main_text, option1, option2, option3, completion_status_html])
                submit_btn.click(reset_radio, outputs=[radio])
                reset_button.click(rater.reset, inputs=[], outputs=[main_text, option1, option2, option3, completion_status_html])
                save_btn.click(rater.save_ratings)
        with gr.Row():
            show_ratings_btn = gr.Button(value="Show Ratings")

        with gr.Row():
            ratings_text = gr.JSON(label="Ratings")
            show_ratings_btn.click(rater.show_ratings, outputs=ratings_text)

    return demo

if __name__ == "__main__":

    # add argument parsing for arguments
    # ratings filename, password, share, etc.
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--ratings_filename", type=str, default=None)
    parser.add_argument("--resume_from_filename", type=str, default=None)
    parser.add_argument("--username", type=str, default="admin")
    parser.add_argument("--password", type=str, default=" ")
    parser.add_argument("--share", action="store_true")
    parser.add_argument("--debug", action="store_true")

    args = parser.parse_args()

    if args.password is None:
        password = os.environ.get("GRADIO_RATING_PASS")
    else:
        password = args.password

    rater = RateTexts(ratings_filename=args.ratings_filename, 
                      resume_from_filename=args.resume_from_filename,
                      debug=args.debug)
    demo = create_demo(rater)
    demo.queue().launch(auth=("admin", password), server_name="0.0.0.0", share=args.share)

Nginx Config

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name <mycustomdomain>;
    return 301 https://$host$request_uri;
}

# HTTPS Configuration
server {
    listen 443 ssl;
    server_name <mycustomdomain>;

    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/<mycustomdomain>/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/<mycustomdomain>/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    # Proxy Settings
    location / {
        proxy_pass http://127.0.0.1:7860/;
        proxy_http_version 1.1;
        proxy_redirect off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Screenshot

UI opens successfully from the custom domain URL.

UI hangs forever failing with a different event id:

n id:

Success when using demo from VM-External-IP:7860 from an external machine:

Logs

(base) keremturgutlu@gradio-webui:~/sxs_ratings_webui$ python sxs_ratings_demo_minimal.py --debug
/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/gradio/blocks.py:506: UserWarning: Theme should be a class loaded from gradio.themes
  warnings.warn("Theme should be a class loaded from gradio.themes")
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
WARNING:  Invalid HTTP request received.
WARNING:  Invalid HTTP request received.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/gradio/routes.py", line 653, in queue_data
    blocks._queue.attach_data(body)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/gradio/queueing.py", line 132, in attach_data
    raise ValueError("Event not found", event_id)
ValueError: ('Event not found', '22e2439d803a42fb8f228c4d8b711574')
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/gradio/routes.py", line 653, in queue_data
    blocks._queue.attach_data(body)
  File "/home/keremturgutlu/miniconda3/lib/python3.11/site-packages/gradio/queueing.py", line 132, in attach_data
    raise ValueError("Event not found", event_id)
ValueError: ('Event not found', 'bff5bde278404186a80f25ead17e9e51')

Keeps repeating the error during hang with different event ids not found error.


### System Info

```shell
Gradio Environment Information:
------------------------------
Operating System: Linux
gradio version: 4.0.2
gradio_client version: 0.7.0

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
altair: 5.1.2
fastapi: 0.104.1
ffmpy: 0.3.1
gradio-client==0.7.0 is not installed.
httpx: 0.25.0
huggingface-hub: 0.18.0
importlib-resources: 6.1.0
jinja2: 3.1.2
markupsafe: 2.1.3
matplotlib: 3.8.1
numpy: 1.26.1
orjson: 3.9.10
packaging: 23.1
pandas: 2.1.2
pillow: 10.1.0
pydantic: 2.4.2
pydub: 0.25.1
python-multipart: 0.0.6
pyyaml: 6.0.1
requests: 2.31.0
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.9.0
typing-extensions: 4.8.0
uvicorn: 0.23.2
websockets: 11.0.3
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.

gradio_client dependencies in your environment:

fsspec: 2023.10.0
httpx: 0.25.0
huggingface-hub: 0.18.0
packaging: 23.1
requests: 2.31.0
typing-extensions: 4.8.0
websockets: 11.0.3

Severity

Blocking usage of gradio

abidlabs commented 1 year ago

Starting to see this error locally as well, @KeremTurgutlu. We'll look into it!

speaknowpotato commented 1 year ago

Starting to see this error locally as well, @KeremTurgutlu. We'll look into it!

Thanks @abidlabs . do you have any progress on this issue? it blocks the usage of gradio 4.x.x.

abidlabs commented 1 year ago

Hi folks. We are looking into it but don't have a solution just yet -- cc @aliabid94

BrunoBosshard commented 1 year ago

I have the same problem since upgrading to Gradio 4. It used to work in previous Gradio versions.

It works locally with "localhost;portnumber", but with NGINX as web server for a secure (https) custom domain, it spins endlessly.

Maybe it requires a change in the NGINX configuration ( https://www.gradio.app/guides/running-gradio-on-your-web-server-with-nginx )?

BrunoBosshard commented 1 year ago

The solution might have been discussed here: https://github.com/gradio-app/gradio/issues/6366 . Quote from @AntoDono:

After hours of debug, it turns out I was just missing proxy_buffering off; in location / 😭😭😭

Thanks, @AntoDono !

SilkePilon commented 1 year ago

having the same bug amt.

ui.queue().launch(server_port=8000, show_api=False, quiet=True)

mojobojo commented 11 months ago

I have been experiencing this bug as well. As I was adding and removing components to my UI I noticed there was a certain threshold of the number of them that caused this to happen. I downgraded to v3.39.0 and this stopped happening.

abidlabs commented 11 months ago

Yeah I think its if there are too many events executing simultaneously. We are working on a fix

abidlabs commented 11 months ago

Should be fixed in 4.9.0!

gradio-app / gradio