Closed leond08 closed 1 year ago
@leond08 Thanks for the ticket!
In order to understand how it is happening can you provide a little more info?
I'm using gunicorn3 latest version I'm using eventlet and gevent for this i'm running my flask application - Flask-SocketIO
I start my background task after a user clicked the button My background task function is to listen to an event, after clicking the "DONE" button the background task must stop then send an emit message to all the users
Had the same issue with aiohttp + gunicorn, observe the same message each time when ctrl + c.
[INFO] Error while closing socket [Errno 9] Bad file descriptor
I don't reproduce it. I suspect your application is closing some fds which create the issue above.
We are experiencing the same issue, only thing is it's only happening on 1 out of 8 containers running in docker swarm.
We have experienced the same issue with 1 out of 9 containers, it looks like related to docker and python3 and gevent.
gunicorn 20.0.4 + aiohttp 3.6.2
Gunicorn is running as dev server:
gunicorn --reload app:make_app --bind localhost:5000 --worker-class aiohttp.GunicornWebWorker --workers 2 --access-logfile -
Almost every Ctrl+C ends with
^C[2020-05-23 21:49:50 +0200] [38524] [INFO] Handling signal: int
Exception ignored when trying to write to the signal wakeup fd:
Exception ignored when trying to write to the signal wakeup fd:
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/unix_events.py", line 42, in _sighandler_noop
def _sighandler_noop(signum, frame):
def _sighandler_noop(signum, frame):
OSError: [Errno 9] Bad file descriptor
OSError: [Errno 9] Bad file descriptor
[2020-05-23 21:49:50 +0200] [38526] [INFO] Worker exiting (pid: 38526)
[2020-05-23 21:49:50 +0200] [38528] [INFO] Worker exiting (pid: 38528)
[2020-05-23 21:49:50 +0200] [38524] [INFO] Shutting down: Master
It does not matter if application handled any request or not.
With Sanic 20.3.0:
^C[2020-05-26 13:24:55 +0200] [27706] [INFO] Handling signal: int
[2020-05-26 13:24:55 +0200] [27769] [INFO] Stopping server: 27769, connections: 0
[2020-05-26 13:24:55 +0200] [27769] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27769] [INFO] Worker exiting (pid: 27769)
[2020-05-26 13:24:55 +0200] [27771] [INFO] Stopping server: 27771, connections: 0
[2020-05-26 13:24:55 +0200] [27771] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-05-26 13:24:55 +0200] [27771] [INFO] Worker exiting (pid: 27771)
[2020-05-26 13:24:55 +0200] [27706] [INFO] Shutting down: Master
Same here with Gunicorn 20.0.4 + Uvicorn 0.11.5 worker class on every Ctrl+C
INFO: [12621] [gunicorn.error] Handling signal: int
INFO: [12635] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO: [12634] [gunicorn.error] Error while closing socket [Errno 9] Bad file descriptor
INFO: [12635] [gunicorn.error] Worker exiting (pid: 12635)
INFO: [12634] [gunicorn.error] Worker exiting (pid: 12634)
INFO: [12621] [gunicorn.error] Shutting down: Master
any example of application? Also which version of Python are we talking about?
Ubuntu 20.04, system provided Python 3.8.2 in virtualenv
Example application: https://github.com/zgoda/newsloop-server/tree/d603a1c10c9e8be3d998f62ccc55dd73f4677115 (with aiohttp) or https://github.com/zgoda/newsloop-server/tree/b2a8a7f09fa9848d0384b51a3f6c0bb037fb459e (with Sanic). Exact gunicorn invocation in my earlier comment.
Differences in output between aiohttp and Sanic make me suspicious that there's something wrong involving workers.
Same issue, python 3.8.0 sanic 19.12.2 gunicorn 20.0.4
Edit: this happens when i run locally on my Mac, but not when running inside a Linux docker, might help you
Hello, I guess this issue https://github.com/benoitc/gunicorn/issues/2064 has the same reasons. We have almost the same errors like in issue, but we use gunicorn - 19.9.0
Im experiencing this too, FastAPI + the latest Gunicorn and uvicorn workers with Python 3.8.5
As soon as I stop using uvicorn (i.e. remove this line from my gunicorn config):
worker_class = "uvicorn.workers.UvicornWorker"
The errors dissappear.
As described above, this happens when stopping Gunicorn with Ctrl+C or sending a graceful kill signal to the PID.
[2020-09-12 11:56:37 +1000] [100390] [INFO] Starting gunicorn 20.0.4
[2020-09-12 11:56:37 +1000] [100390] [INFO] Listening at: http://0.0.0.0:6000 (100390)
[2020-09-12 11:56:37 +1000] [100390] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 11:56:37 +1000] [100392] [INFO] Booting worker with pid: 100392
[2020-09-12 11:56:38 +1000] [100392] [INFO] Started server process [100392]
[2020-09-12 11:56:38 +1000] [100392] [INFO] Waiting for application startup.
[2020-09-12 11:56:38 +1000] [100392] [INFO] Application startup complete.
[2020-09-12 11:56:48 +1000] [100390] [INFO] Handling signal: term
[2020-09-12 11:56:48 +1000] [100392] [INFO] Shutting down
[2020-09-12 11:56:48 +1000] [100392] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 11:56:48 +1000] [100392] [INFO] Waiting for application shutdown.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Application shutdown complete.
[2020-09-12 11:56:48 +1000] [100392] [INFO] Finished server process [100392]
[2020-09-12 11:56:48 +1000] [100392] [INFO] Worker exiting (pid: 100392)
[2020-09-12 11:56:48 +1000] [100390] [INFO] Shutting down: Master
Here's an exact replication of the issue:
[fots@workstation testing]$ python3.8 -V
Python 3.8.5
[fots@workstation testing]$ python3.8 -m venv ~/.virtualenv/testing
[fots@workstation testing]$ source ~/.virtualenv/testing/bin/activate
(testing) [fots@workstation testing]$ pip install fastapi gunicorn uvicorn
Collecting fastapi
Using cached fastapi-0.61.1-py3-none-any.whl (48 kB)
Collecting gunicorn
Using cached gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Collecting uvicorn
Using cached uvicorn-0.11.8-py3-none-any.whl (43 kB)
Collecting pydantic<2.0.0,>=1.0.0
Using cached pydantic-1.6.1-cp38-cp38-manylinux2014_x86_64.whl (11.5 MB)
Collecting starlette==0.13.6
Using cached starlette-0.13.6-py3-none-any.whl (59 kB)
Requirement already satisfied: setuptools>=3.0 in /home/fots/.virtualenv/testing/lib/python3.8/site-packages (from gunicorn) (47.1.0)
Collecting h11<0.10,>=0.8
Using cached h11-0.9.0-py2.py3-none-any.whl (53 kB)
Collecting websockets==8.*
Using cached websockets-8.1-cp38-cp38-manylinux2010_x86_64.whl (78 kB)
Collecting httptools==0.1.*; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
Using cached httptools-0.1.1-cp38-cp38-manylinux1_x86_64.whl (227 kB)
Collecting uvloop>=0.14.0; sys_platform != "win32" and sys_platform != "cygwin" and platform_python_implementation != "PyPy"
Using cached uvloop-0.14.0-cp38-cp38-manylinux2010_x86_64.whl (4.7 MB)
Collecting click==7.*
Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Installing collected packages: pydantic, starlette, fastapi, gunicorn, h11, websockets, httptools, uvloop, click, uvicorn
Successfully installed click-7.1.2 fastapi-0.61.1 gunicorn-20.0.4 h11-0.9.0 httptools-0.1.1 pydantic-1.6.1 starlette-0.13.6 uvicorn-0.11.8 uvloop-0.14.0 websockets-8.1
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/home/fots/.virtualenv/testing/bin/python3.8 -m pip install --upgrade pip' command.
(testing) [fots@workstation testing]$ ls -l
total 4
-rw-rw-r-- 1 fots fots 117 Sep 12 12:13 main.py
(testing) [fots@workstation testing]$ cat main.py
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
async def root():
return {"message": "Hello World"}
(testing) [fots@workstation testing]$ gunicorn -k uvicorn.workers.UvicornWorker main:app
[2020-09-12 12:19:05 +1000] [105788] [INFO] Starting gunicorn 20.0.4
[2020-09-12 12:19:05 +1000] [105788] [INFO] Listening at: http://127.0.0.1:8000 (105788)
[2020-09-12 12:19:05 +1000] [105788] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-09-12 12:19:05 +1000] [105790] [INFO] Booting worker with pid: 105790
[2020-09-12 12:19:05 +1000] [105790] [INFO] Started server process [105790]
[2020-09-12 12:19:05 +1000] [105790] [INFO] Waiting for application startup.
[2020-09-12 12:19:05 +1000] [105790] [INFO] Application startup complete.
^C[2020-09-12 12:19:06 +1000] [105788] [INFO] Handling signal: int
[2020-09-12 12:19:06 +1000] [105790] [INFO] Shutting down
[2020-09-12 12:19:06 +1000] [105790] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2020-09-12 12:19:06 +1000] [105790] [INFO] Waiting for application shutdown.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Application shutdown complete.
[2020-09-12 12:19:06 +1000] [105790] [INFO] Finished server process [105790]
[2020-09-12 12:19:06 +1000] [105790] [INFO] Worker exiting (pid: 105790)
[2020-09-12 12:19:07 +1000] [105788] [INFO] Shutting down: Master
And here's the pip freeze
output:
click==7.1.2
fastapi==0.61.1
gunicorn==20.0.4
h11==0.9.0
httptools==0.1.1
pydantic==1.6.1
starlette==0.13.6
uvicorn==0.11.8
uvloop==0.14.0
websockets==8.1
I attempted to install uvicorn and gunicorn from GitHub (master branch) to ensure I got the very latest fixes but the problem persisted.
Hope this helps Fotis
gunicorn
and uvicorn
. This error message starts occurring with uvicorn==0.11.4
and not the previous version 0.11.3
(both on OSx and in a Linux container). This is consistent with the reports above with uvicorn, where the versions reported are alwasy greater than 0.11.4
. Evidence at the enduvicorn
starlette
and fastapi
on top of the stack gunicorn+uvicorn
as reported above; - runs the latest uvicorn version 12.X
instead of 0.11.4
; - runs gunicorn
with more than just one uvicorn
workerEvidence. In a new folder on osx run the attached script test.sh
(tested on osx). For testing in the Linux container, save both the script and the Dockerfile and then read the header of the Dockerfile. I also attach the log of the script.
@benoitc, what do you think of this commit in uvicorn
that seems to introduce the bug? The problem seems to be at the interface between gunicorn
and uvicorn
. Can you comment on the ordering of the 2 blocks of code changed in the commit mentioned above in uvicorn
? This may help finding out why this happens with the other cases too. So far this has been reported also with aiohttp
, gevent
, Flask-SocketIO
sanic
. I attach also the log of the script for ease.
file test.sh
#!/bin/bash
python3 -m venv venv
source venv/bin/activate
pip install gunicorn==20.0.04 uvicorn==0.11.4
# create simple uvicorn app
printf "async def app(scope, receive, send):\n await send()\n" > example.py
# spin up gunicorn with 1 uvicorn worker, and then send TERM signal to gunicorn
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn
sleep 1
# you will see 1 log entry like this one:
# [XX] [YY] [INFO] Error while closing socket [Errno 9] Bad file descriptor
printf "\n\n[INFO] if you instead bump down uvicorn's version from 11.4 to 11.3 [Errno 9] goes away:\n\n"
pip install uvicorn==0.11.3
gunicorn example:app -w 1 -k uvicorn.workers.UvicornWorker &
sleep 5 && pkill -f gunicorn
file Dockerfile
# run with:
# docker run -it $(docker build -q .)
FROM python:3.8
COPY test.sh .
RUN chmod +x /test.sh
CMD /test.sh
I had exactly same issue. Here is my case.
Brief : I'm trying to establish a test application for Django dwebsocket with gunicorn. When I trying to use websocket_client to test the result, after close the websocket this error happen everytime.
Environment : Docker image : python:3.7 python version : python3.7.6 gunicorn : version = 20.0.4, work = gevent Django version : Django==2.2 dwebsocket version : 0.5.12
Code:
view.py
from dwebsocket import accept_websocket
@accept_websocket
def my_ws(request):
if request.is_websocket():
ws = request.websocket
while True:
msg = ws.wait(timeout=15)
if msg is None:
print('get None message')
break
else:
msg = b'echo :' + msg
ws.send(msg)
print('send ws seccess')
print('websocket close')
urls.py
from websocketInfo.views import my_ws
from django.conf.urls import url
urlpatterns = [
url(r'my_ws/$', my_ws, name='my_ws')
]
websocket_client
from websocket import create_connection
ws = create_connection("ws://127.0.0.1:8080/my_ws/")
print("Sending 'Hello, World'...")
ws.send("Hello, World")
print("Receiving...")
result = ws.recv()
print(result)
ws.close()
print('ws close')
commad to run gunicorn server
gunicorn MyWebsocket.wsgi -b 0.0.0.0:8000 -w 3 -k gevent
error output
send ws seccess
get None message
websocket close
[2021-01-13 02:43:56 +0000] [101] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
super().handle_request(listener_name, req, sock, addr)
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 128, in handle_request
util.reraise(*sys.exc_info())
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 625, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 114, in handle_request
resp.write(item)
File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 326, in write
self.send_headers()
File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 322, in send_headers
util.write(self.sock, util.to_bytestring(header_str, "latin-1"))
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 286, in write
sock.sendall(data)
File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 523, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "/usr/local/lib/python3.7/site-packages/gevent/_socketcommon.py", line 367, in _sendall
chunk_size = max(socket.getsockopt(SOL_SOCKET, SO_SNDBUF), 1024 * 1024)
File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 156, in __getattr__
return getattr(self._sock, name)
File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 66, in _dummy
raise OSError(EBADF, 'Bad file descriptor')
OSError: [Errno 9] Bad file descriptor
@ChrisXiaoShu The stack trace you posted tells us that this specific socket object has been explicitly closed at the Python level (that's when gevent uses its _dummy
to generate the same exceptions that the operating system does). This means some part of your application stack is closing the socket before returning the response to let gunicorn handle it; at the point the error occurs, gunicorn hadn't even sent HTTP response headers yet.
Kind of the same problem in my case, with the difference that this error happens without doing anything. Sometimes after 5min, sometimes after 2hrs...
had same issue on that setup:
Docker image : python:3.7.10-slim-stretch python version : python3.7.10 gunicorn : gunicorn = {version = "20.1.0", extras = ["eventlet"]} Django version : Django = "2.2.18"
I don’t know if the issue is still unresolved! I had a similar issue (FastAPI + uvicorn worker + gunicorn + supervisor). Gunicorn was unable to stop workers and I saw the above error in the log.
I give the unique process name in the gunicorn settings proc_name
and it's working!!
Last time I had this error It was because of a print statement, it raises this error & kill the worker.
Removing the print statement & replacing it with logging works. Hope it helps
I was trying to tackle this issue, since it's happening to us every time Gunicorn terminates an uvicorn
worker which reached the max_requests
limit.
At some point of my research, I ended up reading https://bugs.python.org/issue29343 and I realised that maybe we should just forget about logging.info()
in case of a socket.EBADF
(Bad file descriptor
). It's not an issue per se and some core devs are even proposing to remove this failure, just like it was before.
A second approach would be logging it as DEBUG
instead of INFO
. Let me know if you like any of the above, I could even submit a PR for it.
Similar problem here, k8s, gunicorn with gevent, flask:
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.189Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
listener_name = listener.getsockname()
File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.190Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
listener_name = listener.getsockname()
File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.191Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
listener_name = listener.getsockname()
File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.191Z ERROR gunicorn.error "Socket error processing request." service=ubuntu.com pid=12 errno=EBADF strerror="Bad file descriptor"
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/root/.local/lib/python3.8/site-packages/gunicorn/util.py", line 626, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 38, in handle
listener_name = listener.getsockname()
File "/root/.local/lib/python3.8/site-packages/gevent/_socketcommon.py", line 557, in getsockname
return self._sock.getsockname()
OSError: [Errno 9] Bad file descriptor
2022-01-21 14:10:31.195Z ERROR flask.app "Exception on /security/cve [GET]" service=ubuntu.com pid=12 request_id=e84ae25c8bc81046044c52901f176248
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/root/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/webapp/security/views.py", line 735, in cve_index
return flask.render_template(
File "/root/.local/lib/python3.8/site-packages/flask/templating.py", line 136, in render_template
ctx.app.update_template_context(context)
File "/root/.local/lib/python3.8/site-packages/flask/app.py", line 838, in update_template_context
context.update(func())
File "/srv/webapp/app.py", line 272, in context
"releases": releases(),
File "/srv/webapp/context.py", line 39, in releases
return yaml.load(releases, Loader=yaml.FullLoader)
File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
return loader.get_single_data()
File "/root/.local/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 71, in compose_node
event = self.peek_event()
File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 107, in peek_event
def peek_event(self):
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 202, in handle_abort
self.cfg.worker_abort(self)
File "/root/.local/lib/python3.8/site-packages/talisker/gunicorn.py", line 123, in gunicorn_worker_abort
raise talisker.wsgi.RequestTimeout(
talisker.wsgi.RequestTimeout: gunicorn worker timeout (pid: 12)
2022-01-21 14:10:31.997Z INFO talisker.wsgi "GET /security/cve?" method=GET path=/security/cve qs="package=libcurl4" status=500 view=webapp.security.views.cve_index duration_ms=9994.17 ip=91.189.88.181 proto=HTTP/1.1 length=9295 forwarded="2600:1900:2000:d9::2, 91.189.88.181" ua=Go-http-client/1.1 service=ubuntu.com pid=12 request_id=e84ae25c8bc81046044c52901f176248
2022-01-21 14:10:34.043Z INFO gunicorn.error "Worker exiting (pid: 12)" service=ubuntu.com pid=12
Sentry is attempting to send 126 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
2022-01-21 14:10:44.956Z INFO gunicorn.error "Booting worker with pid: 41"
2022-01-21 14:10:47.595Z INFO talisker.flask "updating raven config from flask app"
It sounds kind of like some people think this is a benign error that shouldn't be logged, but in our case it appears to be killing the worker - so not benign.
It sounds kind of like some people think this is a benign error that shouldn't be logged, but in our case it appears to be killing the worker - so not benign.
That's definitely an issue, but Gunicorn logs the same error as INFO logs somewhere else, and that's the one I was referring to. https://github.com/benoitc/gunicorn/blob/cf55d2cec277f220ebd605989ce78ad1bb553c46/gunicorn/sock.py#L69
I am surprised to see this issue still open. Following the report from @gg349 (https://github.com/benoitc/gunicorn/issues/1877#issuecomment-739526302) above, I've made a fork and inverted the lines here:
And I run smoothly since months... I do not clearly understand the impact of this change, hence I didn't make a PR. But I am now reluctant of going back to the official branch.
This seems to be biting me as well running saleor API deployed with gunicorn + uvicorn worker. The commit from @onekiloparsec kind of makes sense but that commit branch is old enough that its not compatible with my other dependencies when I try to install it. Forking the uvicorn repo to try and cherry pick that commit into my running version to see if it helps.
EDIT: Actually the 0.17.6 branch has the code already in the form of @onekiloparsec commit. I perhaps started seeing this only after adding --max-requests 2
argument to gunicorn.
Latest uvicorn logic:
# Stop accepting new connections.
for server in self.servers:
server.close()
for sock in sockets or []:
sock.close()
for server in self.servers:
await server.wait_closed()
@matteius if you confirm running smoothly for a few days with 0.17.6, I may reconsider going back to the main branch!
i am not reproucingi t myself. closing the issue
I use this FastAPI Docker image, and this error occurs every time I restart my services: https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker
@benoitc Did you try to run the script that @gg349 posted above? https://github.com/benoitc/gunicorn/issues/1877#issuecomment-739526302
I just tested that and it replicates the error. Just create the Dockerfile
and test.sh
scripts with @gg349's code in the comment linked above, and run docker run -it $(docker build -q .)
The test.sh
script runs two test cases, one with the error, and one without (after downgrading uvicorn
).
Here is the relevant output which includes the error (fourth line from the bottom):
[2022-10-14 11:44:55 +0000] [25] [INFO] Starting gunicorn 20.0.4
[2022-10-14 11:44:55 +0000] [25] [INFO] Listening at: http://127.0.0.1:8000 (25)
[2022-10-14 11:44:55 +0000] [25] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2022-10-14 11:44:55 +0000] [28] [INFO] Booting worker with pid: 28
[2022-10-14 11:44:55 +0000] [28] [INFO] Started server process [28]
[2022-10-14 11:44:55 +0000] [28] [INFO] Waiting for application startup.
[2022-10-14 11:44:55 +0000] [28] [INFO] ASGI 'lifespan' protocol appears unsupported.
[2022-10-14 11:44:55 +0000] [28] [INFO] Application startup complete.
[2022-10-14 11:45:00 +0000] [25] [INFO] Handling signal: term
[2022-10-14 11:45:00 +0000] [28] [INFO] Shutting down
[2022-10-14 11:45:00 +0000] [28] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2022-10-14 11:45:00 +0000] [28] [INFO] Finished server process [28]
[2022-10-14 11:45:00 +0000] [28] [INFO] Worker exiting (pid: 28)
[2022-10-14 11:45:00 +0000] [25] [INFO] Shutting down: Master
I think this issue should remain open.
Still happening for me with FastAPI + Gunicorn + Uvicorn in Docker.
@benoitc Maybe I can help you here...
The problem here is that the Arbiter
on stop
, calls the close_sockets
:
https://github.com/benoitc/gunicorn/blob/69c508ac6e4b301045d3ce21acce8b416415d4c5/gunicorn/arbiter.py#L382
The thing is that close_sockets
will unlink
them:
https://github.com/benoitc/gunicorn/blob/69c508ac6e4b301045d3ce21acce8b416415d4c5/gunicorn/sock.py#L211-L212
This decision was made on https://github.com/benoitc/gunicorn/pull/1310. At the time, https://github.com/benoitc/gunicorn/pull/1309 was closed. Maybe it shouldn't have been...
I'll merge https://github.com/encode/uvicorn/pull/1710 on uvicorn
before the next release, which will make this error appear more frequently. That will be merged to solve the forwarding of SIGQUIT
instead of SIGINT
, as we previously discussed on https://github.com/benoitc/gunicorn/issues/2604.
This also happens for me when using Uvicorn worker class. In my case, I run nginx and Gunicorn in a Docker container where nginx server proxies requests to UNIX socket bound by Gunicorn. Both nginx and Gunicorn are managed by Supervisor. I also define a file watch for my app's config file (using watchfiles) in Supervisor's configuration which sends HUP signal to Gunicorn upon a file change. According to the documentation, this should be used for a graceful reload. It works well but exhibits the error with closing the socket.
Here's the components' version info:
Relevant part of supervisord.conf
:
[program:gunicorn]
command = gunicorn -c ./gunicorn.conf.py
stdout_logfile = /dev/stdout
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/stderr
stderr_logfile_maxbytes = 0
[program:config_watch]
command = watchfiles 'supervisorctl -c ./supervisord.conf signal HUP gunicorn' %(ENV_CONFIG_FILE)s
stdout_logfile = /dev/stdout
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/stderr
stderr_logfile_maxbytes = 0
[program:nginx]
command = nginx -g 'daemon off;'
stopsignal = QUIT
stdout_logfile = /dev/stdout
stdout_logfile_maxbytes = 0
stderr_logfile = /dev/stderr
stderr_logfile_maxbytes = 0
CONFIG_FILE
environment variable contains a path to my app's config file.
Relevant part of nginx.conf
:
server {
listen 80 default_server;
server_name _;
error_log stderr;
access_log /dev/stdout;
location / {
gzip on;
gzip_types application/json;
gzip_min_length 1000;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://unix:/tmp/gunicorn.socket;
}
}
gunicorn.conf.py
:
import multiprocessing
user = "www-data"
group = "www-data"
bind = "unix:/tmp/gunicorn.socket"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "uvicorn.workers.UvicornWorker"
worker_tmp_dir = "/dev/shm"
timeout = 120
graceful_timeout = 120
keepalive = 5
accesslog = None
errorlog = "-"
loglevel = "info"
wsgi_app = "app:app"
Here's the Docker container log:
2023-02-07 11:58:11,874 INFO Set uid to user 0 succeeded
2023-02-07 11:58:11,931 INFO RPC interface 'supervisor' initialized
2023-02-07 11:58:11,931 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-02-07 11:58:11,932 INFO supervisord started with pid 1
2023-02-07 11:58:12,938 INFO spawned: 'config_watch' with pid 7
2023-02-07 11:58:12,944 INFO spawned: 'gunicorn' with pid 8
2023-02-07 11:58:12,950 INFO spawned: 'nginx' with pid 9
[11:58:13] watchfiles v0.18.1 👀 path="/srv/app/config.yaml" target="supervisorctl -c ./supervisord.conf signal HUP gunicorn" (command) filter=DefaultFilter...
gunicorn: signalled
2023-02-07 11:58:13,600 WARN exited: gunicorn (terminated by SIGHUP; not expected)
2023-02-07 11:58:14,601 INFO success: config_watch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-02-07 11:58:14,605 INFO spawned: 'gunicorn' with pid 16
2023-02-07 11:58:14,606 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2023-02-07 11:58:15 +0000] [16] [INFO] Starting gunicorn 20.1.0
[2023-02-07 11:58:15 +0000] [16] [INFO] Listening at: unix:/tmp/gunicorn.socket (16)
[2023-02-07 11:58:15 +0000] [16] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2023-02-07 11:58:15 +0000] [17] [INFO] Booting worker with pid: 17
[2023-02-07 11:58:15 +0000] [18] [INFO] Booting worker with pid: 18
[2023-02-07 11:58:15 +0000] [19] [INFO] Booting worker with pid: 19
[2023-02-07 11:58:15 +0000] [20] [INFO] Booting worker with pid: 20
[2023-02-07 11:58:15 +0000] [21] [INFO] Booting worker with pid: 21
[2023-02-07 11:58:15 +0000] [22] [INFO] Booting worker with pid: 22
2023-02-07 11:58:15,689 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2023-02-07 11:58:15 +0000] [23] [INFO] Booting worker with pid: 23
[2023-02-07 11:58:15 +0000] [24] [INFO] Booting worker with pid: 24
[2023-02-07 11:58:15 +0000] [25] [INFO] Booting worker with pid: 25
[2023-02-07 11:58:16 +0000] [18] [INFO] Started server process [18]
[2023-02-07 11:58:16 +0000] [18] [INFO] Waiting for application startup.
[2023-02-07 11:58:16 +0000] [18] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [19] [INFO] Started server process [19]
[2023-02-07 11:58:17 +0000] [19] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [19] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [17] [INFO] Started server process [17]
[2023-02-07 11:58:17 +0000] [17] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [17] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [20] [INFO] Started server process [20]
[2023-02-07 11:58:17 +0000] [20] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [20] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [23] [INFO] Started server process [23]
[2023-02-07 11:58:17 +0000] [23] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [23] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [21] [INFO] Started server process [21]
[2023-02-07 11:58:17 +0000] [21] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [21] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [24] [INFO] Started server process [24]
[2023-02-07 11:58:17 +0000] [24] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [24] [INFO] Application startup complete.
[2023-02-07 11:58:17 +0000] [25] [INFO] Started server process [25]
[2023-02-07 11:58:17 +0000] [25] [INFO] Waiting for application startup.
[2023-02-07 11:58:17 +0000] [25] [INFO] Application startup complete.
[2023-02-07 11:58:18 +0000] [22] [INFO] Started server process [22]
[2023-02-07 11:58:18 +0000] [22] [INFO] Waiting for application startup.
[2023-02-07 11:58:18 +0000] [22] [INFO] Application startup complete.
...modifying the config file now...
[11:58:53] 1 change detected
[11:58:53] process already dead, exit code: 0
gunicorn: signalled
[2023-02-07 11:58:53 +0000] [16] [INFO] Handling signal: hup
[2023-02-07 11:58:53 +0000] [16] [INFO] Hang up: Master
[2023-02-07 11:58:53 +0000] [41] [INFO] Booting worker with pid: 41
[2023-02-07 11:58:53 +0000] [44] [INFO] Booting worker with pid: 44
[2023-02-07 11:58:53 +0000] [43] [INFO] Booting worker with pid: 43
[2023-02-07 11:58:53 +0000] [42] [INFO] Booting worker with pid: 42
[2023-02-07 11:58:53 +0000] [46] [INFO] Booting worker with pid: 46
[2023-02-07 11:58:53 +0000] [45] [INFO] Booting worker with pid: 45
[2023-02-07 11:58:53 +0000] [47] [INFO] Booting worker with pid: 47
[2023-02-07 11:58:53 +0000] [48] [INFO] Booting worker with pid: 48
[2023-02-07 11:58:53 +0000] [22] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [22] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [17] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [17] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [49] [INFO] Booting worker with pid: 49
[2023-02-07 11:58:53 +0000] [20] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [20] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [21] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [21] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [24] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [24] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [25] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [25] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [18] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [18] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [23] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [23] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [19] [INFO] Shutting down
[2023-02-07 11:58:53 +0000] [19] [INFO] Error while closing socket [Errno 9] Bad file descriptor
[2023-02-07 11:58:53 +0000] [22] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [22] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [22] [INFO] Finished server process [22]
[2023-02-07 11:58:53 +0000] [22] [INFO] Worker exiting (pid: 22)
[2023-02-07 11:58:53 +0000] [17] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [17] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [17] [INFO] Finished server process [17]
[2023-02-07 11:58:53 +0000] [17] [INFO] Worker exiting (pid: 17)
[2023-02-07 11:58:53 +0000] [20] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [20] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [20] [INFO] Finished server process [20]
[2023-02-07 11:58:53 +0000] [21] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [20] [INFO] Worker exiting (pid: 20)
[2023-02-07 11:58:53 +0000] [21] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [21] [INFO] Finished server process [21]
[2023-02-07 11:58:53 +0000] [21] [INFO] Worker exiting (pid: 21)
[2023-02-07 11:58:53 +0000] [24] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [24] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [24] [INFO] Finished server process [24]
[2023-02-07 11:58:53 +0000] [24] [INFO] Worker exiting (pid: 24)
[2023-02-07 11:58:53 +0000] [25] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [25] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [25] [INFO] Finished server process [25]
[2023-02-07 11:58:53 +0000] [25] [INFO] Worker exiting (pid: 25)
[2023-02-07 11:58:53 +0000] [18] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [18] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [18] [INFO] Finished server process [18]
[2023-02-07 11:58:53 +0000] [18] [INFO] Worker exiting (pid: 18)
[2023-02-07 11:58:53 +0000] [23] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [23] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [23] [INFO] Finished server process [23]
[2023-02-07 11:58:53 +0000] [23] [INFO] Worker exiting (pid: 23)
[2023-02-07 11:58:53 +0000] [19] [INFO] Waiting for application shutdown.
[2023-02-07 11:58:53 +0000] [19] [INFO] Application shutdown complete.
[2023-02-07 11:58:53 +0000] [19] [INFO] Finished server process [19]
[2023-02-07 11:58:53 +0000] [19] [INFO] Worker exiting (pid: 19)
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 20 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 22 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 23 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 25 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 21 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 17 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 18 was terminated due to signal 15
[2023-02-07 11:58:54 +0000] [16] [WARNING] Worker with pid 24 was terminated due to signal 15
[2023-02-07 11:58:55 +0000] [41] [INFO] Started server process [41]
[2023-02-07 11:58:55 +0000] [41] [INFO] Waiting for application startup.
[2023-02-07 11:58:55 +0000] [41] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [45] [INFO] Started server process [45]
[2023-02-07 11:58:56 +0000] [45] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [45] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [42] [INFO] Started server process [42]
[2023-02-07 11:58:56 +0000] [42] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [42] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [43] [INFO] Started server process [43]
[2023-02-07 11:58:56 +0000] [43] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [43] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [49] [INFO] Started server process [49]
[2023-02-07 11:58:56 +0000] [49] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [49] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [47] [INFO] Started server process [47]
[2023-02-07 11:58:56 +0000] [47] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [47] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [48] [INFO] Started server process [48]
[2023-02-07 11:58:56 +0000] [48] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [44] [INFO] Started server process [44]
[2023-02-07 11:58:56 +0000] [44] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [48] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [44] [INFO] Application startup complete.
[2023-02-07 11:58:56 +0000] [46] [INFO] Started server process [46]
[2023-02-07 11:58:56 +0000] [46] [INFO] Waiting for application startup.
[2023-02-07 11:58:56 +0000] [46] [INFO] Application startup complete.
I also see this error on latest gunicorn+uvicorn every time max-requests triggers and reaps a worker.
For us the problem was our form was doing a POST /api
while the route was on /api/
. 🤦
The trailing slash was resulting in a 307 redirect and... making the file not write?
What is the solution to above issue?
Gunicorn (20.1.0) + Uvicorn (0.22.0) -> same issue
This happens for me, too.
Happens to me too :
@Aniket-Singla please upgrade to 21.2.0 this is fixing such issue. Thanks for the report, but next time can you open a new ticket or create a new one that link to a closed issue. This is described in the contributing document and helps me to track issues )
@Aniket-Singla please upgrade to 21.2.0 this is fixing such issue. Thanks for the report, but next time can you open a new ticket or create a new one that link to a closed issue. This is described in the contributing document and helps me to track issues )
Thankyou so much for your reply and sorry for not reading the contributing Doc. Will upgrade
It's a uvloop issue. Setting Uvicorn's loop to asyncio
(by overriding uvicorn.workers.UvicornWorker.CONFIG_KWARGS
) fixed it for me.
The same issue to me, but it's only occurred on my k8s deployment.
Here is my repo, make k
to deploy it by k8s to reproduce the issue.
The same for me with gunicorn == 21.2.0 and meinheld == 1.0.1.
I'm getting this Error while closing socket [Errno 9] Bad file descriptor
on 21.2.0. Was there any workaround or anything? I'm using FastAPI.
I'm running gunicorn in production with systemd. I periodically get [INFO] Handling signal: term
and then Error while closing socket [Errno 9] Bad file descriptor
and then it won't start up again saying Connection in use: ('127.0.0.1', 8000)
. This has been driving me nuts because prod goes down. Any help or pointers are appreciated.
And I checked memory, that's not it.
It's a uvloop issue. Setting Uvicorn's loop to
asyncio
(by overridinguvicorn.workers.UvicornWorker.CONFIG_KWARGS
) fixed it for me.
can you explain more about this? my server is getting full of these logs
It's a uvloop issue. Setting Uvicorn's loop to
asyncio
(by overridinguvicorn.workers.UvicornWorker.CONFIG_KWARGS
) fixed it for me.can you explain more about this? my server is getting full of these logs
So I just tried this and it did work for me. Here's a minimal example for FastAPI:
I put this into uvicorn_utils.py
in my fast_api_app
folder.
from uvicorn.workers import UvicornWorker
MyUvicornWorker = UvicornWorker
MyUvicornWorker.CONFIG_KWARGS = {"loop": "asyncio", "http": "auto"}
I use a gunicorn.conf.py
file which then I invoke with gunicorn fast_api_app.main:app -c gunicorn.conf.py
with the following
worker_class = "fast_api_app.uvicorn_utils.MyUvicornWorker"
Now I no longer get file descriptor warnings. I'm using gunicorn 20.1.0, uvicorn 0.24.0.post1 and fastapi 0.108.0.
I'm having this issue in raspbian image
[2018-09-10 20:40:11 +0800] [21421] [CRITICAL] WORKER TIMEOUT (pid:21699) [2018-09-10 20:40:11 +0800] [21699] [ERROR] Socket error processing request. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/gunicorn/workers/async.py", line 62, in handle six.reraise(*sys.exc_info()) File "/usr/lib/python3/dist-packages/gunicorn/six.py", line 625, in reraise raise value File "/usr/lib/python3/dist-packages/gunicorn/workers/async.py", line 35, in handle listener_name = listener.getsockname() OSError: [Errno 9] Bad file descriptor