openfaas / of-watchdog

Reverse proxy for STDIO and HTTP microservices
MIT License
259 stars 115 forks source link

Upstream HTTP request error on gunicorn worker auto-restart #102

Open lihopentium opened 3 years ago

lihopentium commented 3 years ago

We use gunicorn on fprocess command ENV fprocess="gunicorn -c gconfig.py index:app" The of-watchdog appear the Upstream HTTP request error while the gunicon worker restart by the max_requests setting.

I use ab to stress test on of-watchdog and gunicorn endpoint, and only happened on the POST method. The of-watchdog endpoint had some failed requests while the gunicorn worker restart. But direct sending requests to gunicorn endpoint seems no problem.

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

code:

gunicorn config

bind = "0.0.0.0:5000"
workers = 4
worker_class = "gthread"
threads = 1
timeout = 300
graceful_timeout = 30
loglevel = "debug"
max_requests = 10

Steps

  1. run the docker container
    docker run -it -p 8080:8080 -p 5000:5000 lihodocker/faas-python3-gunicorn:20200710182115
  2. create the body file
    echo "{}" > /tmp/body.txt
  3. stress test on gunicorn endpoint, increase the concurrent requests or total requests as you want
    ab -c 20 -n 100 -p /tmp/body.txt http://localhost:8080/

    you may see some failed requests due to the gunicorn worker auto-restart.

failed log

2020/07/10 10:36:03 Upstream HTTP request error: Post http://127.0.0.1:5000/: read tcp 127.0.0.1:57600->127.0.0.1:5000: read: connection reset by peer
  1. stress test on gunicorn endpoint, increase the concurrent requests or total requests as you want

    ab -c 20 -n 100 -p /tmp/body.txt http://localhost:5000/

    There are no failed requests on gunicorn endpoint.

  2. Remove the max_requests = 10 setting from gconfig.py, run step 3 again. No restarts on gunicorn workers, requests are all success.

Context

of-watchdog have the retry or fix the connection close as well as other HTTP clients do.

Your Environment

lihopentium commented 3 years ago

I have a solution proposal, could we make the keep-alive to be an optional argument. Let the user decide to turn on or turn off for their use cases. @alexellis Does it make sense to you?