Increase `httpx` timeout to prevent requests from erroring out

[chintan-donda]

Describe the bug

I'm developing a chatbot based on the RAG-QA.

I'm getting an Error on the Gradio front-end even though the backend works properly.

I'm uploading 1 page from Wikipedia using LangChain WikipediaLoader. Then creating a vector db using FAISS.

I've enabled logging and it looks as below:

2023-08-09 07:40:14 INFO [vector_store] Creating index...
2023-08-09 07:40:14 WARNING [langchain.text_splitter] Created a chunk of size 2518, which is longer than the specified 500
2023-08-09 07:40:14 WARNING [langchain.text_splitter] Created a chunk of size 793, which is longer than the specified 500
2023-08-09 07:40:14 WARNING [langchain.text_splitter] Created a chunk of size 2212, which is longer than the specified 500
2023-08-09 07:40:14 WARNING [langchain.text_splitter] Created a chunk of size 790, which is longer than the specified 500
Batches: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.90s/it]
2023-08-09 07:40:18 INFO [faiss.loader] Loading faiss.
2023-08-09 07:40:18 INFO [faiss.loader] Successfully loaded faiss.
2023-08-09 07:40:18 INFO [vector_store] Index created successfully!
2023-08-09 07:40:18 INFO [common_utils] create_index took: 3.96 seconds
2023-08-09 07:40:18 INFO [vector_store] wikipedia data uploaded successfully!

In total it takes 3.96 seconds for the index creation.

I'm getting the Error on the Gradio front-end where I'm displaying the Status of the upload if it's successful or not in the gr.Textbox()

Any help please?

Have you searched existing issues? 🔎

• [x] I have searched and found no existing issues

Reproduction

Create the upload button as below:

with gr.Blocks(title="Demo", theme=gr.themes.Soft()) as demo:
        with gr.Column(scale=1):
                upload_success = gr.Textbox(
                    label="Success of the Upload",
                    placeholder="Success status for the data upload would appear here",
                    interactive=False,
                    queue=False
                )

              page = gr.Textbox(
                  label="Enter Wikipedia page names",
                  placeholder="Type the Wikipedia page names here",
                  value="2022 FIFA World Cup"
              )

              b_page = gr.Button("Load Wikipedia Pages")
              b_page.click(
                  fn=load_wiki_pages,
                  inputs=page,
                  outputs=upload_success
              )

demo.queue().launch(share=True)

NOTE: I need to use demo.queue() because I'm developing a RAG-QA Chatbot. Inside chatbot, I'm streaming the response with:

for char in response:
      history[-1][1] += char
      time.sleep(0.03)
      yield history

Loading and creating the vector store as below:

from langchain.document_loaders import WikipediaLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings

docs = WikipediaLoader(query=page, load_max_docs=1).load()

text_splitter = CharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=100,
        separator="\n",
        )

documents = text_splitter.split_documents(docs)

# Vector store using Facebook AI Similarity Search
embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
index = FAISS.from_documents(documents, embeddings)

Screenshot

Originally it looks as below:

On uploading and creating the index from Wikipedia page:

Logs

No response

System Info

gradio                        3.27.0
gradio_client                 0.3.0
langchain                     0.0.228
wikipedia                     1.4.0

### Severity

Blocking usage of gradio

[abidlabs]

Hi @chintan-donda we'd like to help with this issue, but there's a lot of code here that we can't use, such as calls to proprietary APIs. Can you put together a self-contained mock example we can use to debug?

[chintan-donda]

@abidlabs Yes, I'm using the proprietary APIs. Not sure if the issue would be reproducible on your Env or not. As I see from the issues on Gradio github repo, others are also facing the same issue. I've included most of the code.

[freddyaboulton]

Hi @chintan-donda - agree that we would need a self contained repro to be able to debug this. Can you try upgrading your gradio version to see?

[chintan-donda]

@freddyaboulton Same issue with the latest gradio version == 3.40.1

[abidlabs]

Hi @chintan-donda sorry, we'll need a self-contained reproducible example in order to look into this. I'll close this for now, but we can reopen if we get one.

[chintan-donda]

@abidlabs Please find below the self-contained reproducible example.

FYI, I want to do the zero-shot-classification using facebook/bart-large-mnli model.

import gradio as gr
import time
from transformers import pipeline

import logging
logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S"
)
logger = logging.getLogger(__name__)

import warnings
warnings.filterwarnings('ignore')

class DomState:
    def __init__(
        self
    ):
        self.QUERY_TYPES = [
            "Payment done",
            "Update name",
            "Travel plan",
            "Date change",
            "Bank account details",
            "Cancel trip",
            "Refund",
            "Hotel booking",
            "Flights booking",
        ]

        self.MOST_FREQUENT_QUERIES = [
            ["Payment done"],
            ["Update my name in the booking"],
            ["How can I plan the travel to India?"],
            ["I want to change the Date of travel. Please help."],
            ["Where can I enter my nank account details?"],
            ["I want to cancel my trip"],
            ["When will I get my refund?"],
            ["Hotel booking is done. What's next?"],
            ["Help me do flights booking"],
        ]

    def clear_history(self):
        return ["", [("", ""),]]

    def detect_query_intent(self, query):
        logger.info(f"Detecting intent/type/category of the query...")

        query_intent = ""

        classifier = pipeline(
            "zero-shot-classification",
            model="facebook/bart-large-mnli",
        )

        query_intent = classifier(query, self.QUERY_TYPES)
        logger.info(f"query_intent: {query_intent}")
        query_intent = query_intent["labels"][0]

        logger.info(f"Intent of the query: {query_intent}")
        return query_intent

    def submit_bot_query(self, query, history):
        history = history + [[query, ""]]
        return ["", history]

    # Method for query submission from textbox
    def get_bot_response(self, history):
        query = history[-1][0]
        logger.info(f"Query: {query}")

        # Detect intent of the query
        response = self.detect_query_intent(query)
        logger.info(f"Response: {response}")

        history[-1][1] = ""
        # Stream response like ChatGPT
        for char in response:
            history[-1][1] += char
            time.sleep(0.01)
            yield history

    # Method for query submission on clicking examples
    def click_handler_for_most_frequent_query_selection(self, query, history):
        query, history = self.submit_bot_query(query, history)

        query = history[-1][0]
        logger.info(f"Query: {query}")

        # Detect intent of the query
        response = self.detect_query_intent(query)
        logger.info(f"Response: {response}")

        history[-1][1] = ""
        # Stream response like ChatGPT
        for char in response:
            history[-1][1] += char
            time.sleep(0.01)
            yield ["", history]

with gr.Blocks(title='Demo', theme=gr.themes.Soft()) as demo:
    dom = DomState()

    chatbot = gr.Chatbot().style(height=600)

    query = gr.Textbox(
        label="Query",
        placeholder="Type your query here and press enter",
    ).style(container=False)

    # Show most frequently asked queries
    gr.Examples(
        label="Most frequently asked queries",
        examples=dom.MOST_FREQUENT_QUERIES,
        fn=dom.click_handler_for_most_frequent_query_selection,
        inputs=[query, chatbot],
        outputs=[query, chatbot],
        cache_examples=False,
        run_on_click=True,
        examples_per_page=20,
    )

    # Text query submission
    query.submit(
        dom.submit_bot_query,
        [query, chatbot],
        [query, chatbot],
    ).then(
        dom.get_bot_response,
        chatbot,
        chatbot
    )

    # Clear history
    clear_history = gr.Button(value="🗑️  Clear history", interactive=True)
    clear_history.click(
        dom.clear_history,
        [],
        [query, chatbot]
    )

demo.queue().launch(
    server_name="0.0.0.0",
    share=True
)

Below is my system details:

gradio                        3.40.0
gradio_client                 0.4.0
transformers                     4.28.1
torch                         2.0.0

Let me know if you need anything else.

[abidlabs]

Simpler repro provided here: https://github.com/gradio-app/gradio/issues/5273

[chintan-donda]

@abidlabs Please find the simple version:

import gradio as gr
import time

import logging
logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S"
)
logger = logging.getLogger(__name__)

import warnings
warnings.filterwarnings('ignore')

class DomState:
    def __init__(
        self
    ):
        pass

    def clear_history(self):
        return ["", [("", ""),]]

    def submit_bot_query(self, query, history):
        history = history + [[query, ""]]
        return ["", history]

    def get_bot_response(self, history):
        query = history[-1][0]
        logger.info(f"Query: {query}")

        time.sleep(5)         # if I remove this line, it works fine. But with 5 seconds time sleep, Gradio give Error on front-end.
        history[-1][1] = ""
        # Stream response like ChatGPT
        for char in query:
            history[-1][1] += char
            time.sleep(0.05)
            yield history

with gr.Blocks(title='Demo', theme=gr.themes.Soft()) as demo:
    dom = DomState()

    chatbot = gr.Chatbot().style(height=600)

    query = gr.Textbox(
        label="Query",
        placeholder="Type your query here and press enter",
    ).style(container=False)

    # Text query submission
    query.submit(
        dom.submit_bot_query,
        [query, chatbot],
        [query, chatbot],
    ).then(
        dom.get_bot_response,
        chatbot,
        chatbot
    )

    # Clear history
    clear_history = gr.Button(value="🗑️  Clear history", interactive=True)
    clear_history.click(
        dom.clear_history,
        [],
        [query, chatbot]
    )

demo.queue().launch(
    server_name="0.0.0.0",
    share=True
)

System requirements:

gradio                        3.40.0
gradio_client                 0.4.0
Python                        3.9.6

NOTE: time.sleep(5) # if I remove this line, it works fine. But with 5 seconds time sleep, Gradio give Error on front-end.

[abidlabs]

Hi @chintan-donda I can't reproduce this issue locally or in Colab using the latest version of gradio. Here's a link to the working Colab: https://colab.research.google.com/drive/1oR75qc4XRMjkD8vH58B_ogSOuOOPk1QA?usp=sharing

It might be something specific to your local environment causing this issue

[chintan-donda]

Hi @chintan-donda I can't reproduce this issue locally or in Colab using the latest version of gradio. Here's a link to the working Colab: https://colab.research.google.com/drive/1oR75qc4XRMjkD8vH58B_ogSOuOOPk1QA?usp=sharing

It might be something specific to your local environment causing this issue

I've tried in 3 different setups. My local machine, teammate's machine, and VM. All of these machines, I'm getting the same error.

[chintan-donda]

@abidlabs Issue was with one of the lib dependencies httpx. httpx lib has a default timeout=5.0 seconds. ==> this was causing the timeout issue for Gradio, which was showing Error on the Gradio front-end.

Thought to share it here. It'd be useful for others.

[abidlabs]

Interesting thanks @chintan-donda so what did you need to do to fix this?

[chintan-donda]

Interesting thanks @chintan-donda so what did you need to do to fix this?

Manually change the value in this line to higher number, say 30. DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=30.0)

[abidlabs]

Interesting, thanks for letting us know. We can potentially increase the timeout from within Gradio so let me reopen this issue so that we can investigate this further.

[chintan-donda]

Interesting, thanks for letting us know. We can potentially increase the timeout from within Gradio so let me reopen this issue so that we can investigate this further.

That would be of great help. Instead of doing it manually, we'd fix it within Gradio.

[chintan-donda]

@abidlabs You can reproduce the issue with below code:

import gradio as gr
from googletrans import Translator
import time

import logging
logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S"
)
logger = logging.getLogger(__name__)

import warnings
warnings.filterwarnings('ignore')

class DomState:
    def __init__(
        self
    ):
        pass

    def clear_history(self):
        return ["", [("", ""),]]

    def submit_bot_query(self, query, history):
        history = history + [[query, ""]]
        return ["", history]

    def get_bot_response(self, history):
        query = history[-1][0]
        logger.info(f"Query: {query}")

        time.sleep(7)         # if I remove this line, it works fine. But with 5 seconds time sleep, Gradio give Error on front-end.
        history[-1][1] = ""
        # Stream response like ChatGPT
        for char in query:
            history[-1][1] += char
            time.sleep(0.05)
            yield history

with gr.Blocks(title='Demo', theme=gr.themes.Soft()) as demo:
    dom = DomState()

    chatbot = gr.Chatbot().style(height=600)

    query = gr.Textbox(
        label="Query",
        placeholder="Type your query here and press enter",
    ).style(container=False)

    # Text query submission
    query.submit(
        dom.submit_bot_query,
        [query, chatbot],
        [query, chatbot],
    ).then(
        dom.get_bot_response,
        chatbot,
        chatbot
    )

    # Clear history
    clear_history = gr.Button(value="🗑️  Clear history", interactive=True)
    clear_history.click(
        dom.clear_history,
        [],
        [query, chatbot]
    )

demo.queue().launch(
    server_name="0.0.0.0",
    share=True
)

System requirement:

gradio==3.40.0
gradio_client==0.4.0
googletrans==3.0.0

No major change. Just install googletrans==3.0.0 and just import googletrans at the top. And you will see the issue.

To make it work, try changing the timeout value manually inside httpx config to, say, 30.0 in this line. Exact path to the config file is: ~/path-to-venv/lib/python3.9/site-packages/httpx/_config.py ==> line #369

[petroslk]

Hey, I am running a gradio instance within a docker container and query it via the API from the host machine.

This exact issue seems to be a problem with the Client class imported from gradio_client. By running a simple API call as showed in the simple tutorial, all my request time out after 5s.

from gradio_client import Client

client = Client("http://localhost:7860/")
result = client.predict(
        message="Hello!",
        api_name="/chat"
)
print(result)

I edited the same line in the _config.py file of httpx as suggested by @chintan-donda and it works. Posting here since this seems to be the exact same issue

Package Version
------- -------
Brotli  1.0.9
gradio  4.28.3
jupyter 1.0.0
pip     23.3.1
PyQt5   5.15.10
PySocks 1.7.1
sip     6.7.12
wheel   0.41.2

Thanks a lot!

[djaym7]

httpx.ReadTimeout: timed out Tencent/HunyuanDiT#26

+1 using docker container and getting same timeout error

[piwawa]

Interesting, thanks for letting us know. We can potentially increase the timeout from within Gradio so let me reopen this issue so that we can investigate this further.

How to increase the timeout from within Gradio? Any optional parameters?

[djaym7]

@piwawa you can use this script to change the httpx timeout:

#!/bin/bash

# Get the path to the _config.py file
config_path=$(python -c "import httpx; import os; print(os.path.join(os.path.dirname(httpx.__file__), '_config.py'))")

# Check if the file exists
if [ ! -f "$config_path" ]; then
    echo "Error: $config_path does not exist."
    exit 1
fi

# Create a backup of the original file
cp "$config_path" "${config_path}.bak"

# Modify the file
sed -i "s/DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0)/DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=60)/" "$config_path"

# Check if the modification was successful
if grep -q "DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=60)" "$config_path"; then
    echo "Successfully modified $config_path"
    echo "A backup of the original file was created at ${config_path}.bak"
else
    echo "Failed to modify $config_path"
    echo "The original file remains unchanged"
fi

[piwawa]

@piwawa you can use this script to change the httpx timeout:

#!/bin/bash

# Get the path to the _config.py file
config_path=$(python -c "import httpx; import os; print(os.path.join(os.path.dirname(httpx.__file__), '_config.py'))")

# Check if the file exists
if [ ! -f "$config_path" ]; then
    echo "Error: $config_path does not exist."
    exit 1
fi

# Create a backup of the original file
cp "$config_path" "${config_path}.bak"

# Modify the file
sed -i "s/DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0)/DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=60)/" "$config_path"

# Check if the modification was successful
if grep -q "DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=60)" "$config_path"; then
    echo "Successfully modified $config_path"
    echo "A backup of the original file was created at ${config_path}.bak"
else
    echo "Failed to modify $config_path"
    echo "The original file remains unchanged"
fi

Have done this, still report the same error.

Running on local URL:  http://0.0.0.0:8001

To create a public link, set `share=True` in `launch()`.
Exception in thread Thread-3 (_do_normal_analytics_request):
Traceback (most recent call last):
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_sync/http_proxy.py", line 317, in handle_request
    stream = stream.start_tls(**kwargs)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 383, in start_tls
    return self._stream.start_tls(ssl_context, server_hostname, timeout)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 152, in start_tls
    with map_exceptions(exc_map):
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: _ssl.c:990: The handshake operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/gradio/analytics.py", line 70, in _do_normal_analytics_request
    data["ip_address"] = get_local_ip_address()
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/gradio/analytics.py", line 131, in get_local_ip_address
    ip_address = httpx.get(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_api.py", line 198, in get
    return request(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_api.py", line 106, in request
    return client.request(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/bingo06/miniconda3/envs/cu118/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: _ssl.c:990: The handshake operation timed out 

[djaym7]

Turn off gradio analytics, set the environ variable