reload does not work for uvicorn.workers.UvicornWorker

[sandys]

this is my commandline gunicorn -k uvicorn.workers.UvicornWorker --reload-engine=inotify --workers=1 --log-level debug --access-logfile - --error-logfile - --disable-redirect-access-to-syslog --reload main:app

note that i have tested with both master branch of gunicorn as well as last stable release.

This is the code im using

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def read_root():
    # raise 1
    return {"Hello": "World"}

@app.get("/items/{item_id}")
async def read_item(item_id: int, q: str = None):

    return {"item_id 2": item_id, "q": q}

reload does not pickup when i make changes to the app. im using vscode as an editor

[Andrew-Chen-Wang]

You should just use uvicorn itself rather than adding gunicorn on top of it for local development. Something like this:

uvicorn main:app --reload

And then if you have watchgod installed, uvicorn should automatically pick that up to make the reloading process better. If you're on Docker, add --host 0.0.0.0

[systemime]

This is my start command gunicorn my_pro.asgi:application -b unix:/run/day01/gunicorn.socket --reload -w 2 -t 1 -k uvicorn.workers.UvicornWorker, It can be reloaded normally, but there seems to be a 1-2 second delay， @sandys

[tobias-hd]

I've observed similar behavior as @sandys .

The workaround proposed by @Andrew-Chen-Wang is not always useful (not close to production setup; problems with Windows + docker).

@sandys , @systemime : Which python versions, and which OS were you using?

[Andrew-Chen-Wang]

@tobias-hd This is not my workaround; it is the recommended method by uvicorn's documentation: https://www.uvicorn.org/deployment/

Run gunicorn's CLI during deployment; run uvicorn's during development.

If you're using --reload and DEBUG log level, I'm assuming you're developing locally. If you're acting for production as I'm assuming tobias is saying, then you should use gunicorn and do not use the reload flag (use supervisord).

Additionally, in your IDE, you should do CTRL + S or save the file with some hotkeys. That's usually how flask, Django, or uvicorn works/checks for new file changes.

[cochiseruhulessin]

This is my start command gunicorn my_pro.asgi:application -b unix:/run/day01/gunicorn.socket --reload -w 2 -t 1 -k uvicorn.workers.UvicornWorker, It can be reloaded normally, but there seems to be a 1-2 second delay， @sandys

Thats because the -t argument causes the idle workers to reload every second.

[omarsumadi]

Is there any chance this could be revisited? I'm having some problems with Uvicorn in which Migrations are loaded on app startup, triggering django.core.exceptions.SynchronousOnlyOperation. Running Gunicorn of course solves this as we are now using workers rather than threads, but I would really like to have an option to reload.

By any means, what's the problem with having your development server try to be as close as possible to your production server?

For instance, if you try to use Django Prometheus with Uvicorn without Gunicorn, you'll get a trace like this (also using the Django Cookie Cutter Library @Andrew-Chen-Wang :

django            | Process SpawnProcess-1:
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
django            |     self.run()
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
django            |     self._target(*self._args, **self._kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/subprocess.py", line 61, in subprocess_started
django            |     target(sockets=sockets)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 47, in run
django            |     loop.run_until_complete(self.serve(sockets=sockets))
django            |   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 54, in serve
django            |     config.load()
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 306, in load
django            |     self.loaded_app = import_from_string(self.app)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
django            |     module = importlib.import_module(module_str)
django            |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django            |     return _bootstrap._gcd_import(name[level:], package, level)
django            |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
django            |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
django            |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
django            |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
django            |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
django            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django            |   File "./config/asgi.py", line 23, in <module>
django            |     django.setup(set_prefix=False)
django            |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
django            |     apps.populate(settings.INSTALLED_APPS)
django            |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
django            |     app_config.ready()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/apps.py", line 23, in ready
django            |     ExportMigrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/migrations.py", line 52, in ExportMigrations
django            |     executor = MigrationExecutor(connections[alias])
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
django            |     self.loader = MigrationLoader(self.connection)
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 49, in __init__
django            |     self.build_graph()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 212, in build_graph
django            |     self.applied_migrations = recorder.applied_migrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 76, in applied_migrations
django            |     if self.has_table():
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 56, in has_table
django            |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
django            |   File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
django            |     raise SynchronousOnlyOperation(message)
django            | django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

I have no easy access to modify the library.

[benoitc]

I guess this has to do with asyncio event loop interfering with the rest. I will have a look on it.

[omarsumadi]

@benoitc Someone on Stackoverflow actually had the fix to my problem exactly, but it means I needed to add my Project Root to my path which... is something I didn't do previously and I'm looking into the repercussions of doing that.

Here's the link: https://stackoverflow.com/questions/64306187/how-to-make-django-3-channels-and-uvicorn-work-together

[Andrew-Chen-Wang]

@omarsumadi To run migrations, use the manage.py command python manage.py makemigrations && python manage.py migrate. To run the server: uvicorn config.asgi:application --host 0.0.0.0 --reload. The --host 0.0.0.0 allows your mobile device (e.g. like a phone) to access your web server too. Additionally, in your IDE, you should do CTRL + S or save the file with some hotkeys. That's usually how flask, Django, or uvicorn works/checks for new file changes.

Really, just follow uvicorn's documentation. The comment Tobias said about Windows + Docker being an issue is not an issue if you just use uvicorn without docker in the first place...

By any means, what's the problem with having your development server try to be as close as possible to your production server?

It's just the settings. The differences? Default log levels, setting up HTTPS settings, django debug toolbar, DEBUG=False, etc.. Otherwise, yea your dev server shouldn't deviate too much from your production server (and that's also down to personal opinions).

[sandys]

Hi Andrew Thanks for the reply. I know you're trying to be helpful, but most of working on production apps would like 1) docker and 2) gunicorn so that dev is close to production.

All of us here working in large teams have faced the common issues of mac (intel vs m1) vs windows vs linux.

Trust me when I say this - we would pay to get production consistency on dev... versus choosing the convenient option. 🙏

On Wed, 3 Feb, 2021, 21:24 Andrew Chen Wang, notifications@github.com wrote:

@omarsumadi https://github.com/omarsumadi To run migrations, use the manage.py command python manage.py makemigrations && python manage.py migrate. To run the server: uvicorn config.asgi:application --host 0.0.0.0 --reload. The --host 0.0.0.0 allows your mobile device (e.g. like a phone) to access your web server too. Additionally, in your IDE, you should do CTRL + S or save the file with some hotkeys. That's usually how flask, Django, or uvicorn works/checks for new file changes.

Really, just follow uvicorn's documentation. The comment Tobias said about Windows + Docker being an issue is not an issue if you just use uvicorn without docker in the first place...

By any means, what's the problem with having your development server try to be as close as possible to your production server?

It's just the settings. The differences? Default log levels, setting up HTTPS settings, django debug toolbar, DEBUG=False, etc.. Otherwise, yea your dev server shouldn't deviate too much from your production server (and that's also down to personal opinions).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/benoitc/gunicorn/issues/2339#issuecomment-772613108, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYUY4YKMKAGBQTLCR6OTS5FWSJANCNFSM4NKBSMLQ .

[Andrew-Chen-Wang]

Hey Sandy! I love Docker, but some people might not (addressing Tobias' early comment). The commands I just posted work the same way in Docker and with a virtualenv, regardless.

we would pay to get production consistency on dev.

ik I feel the pain as well... But yea Docker: super useful and needed, regardless if you're in a team or not.

...gunicorn so that dev is close to production.

I would still recommend that you use uvicorn itself when working in the Dev environment. It's close enough to gunicorn, and it'll be a little easier on your computer (at least for me) in terms of resource consumption (and noise reduction due to a phenomenal fan). But yea I agree: dev environment shouldn't deviate too much from prod, if at all.

[omarsumadi]

@Andrew-Chen-Wang Hey Andrew - As you said it might be a Docker Problem, whether or not it is, I did put in an Enhancement on the Django Cookie Cutter repo to fix Uvicorn spawning this error for the Docker version - run Uvicorn programmatically. Also using VSCode, so yes everything migrations and watching for file changes works perfectly. The problem is that Uvicorn is not working with this - not a problem with following documentation.

Still, it's sort of an interesting perspective because I did have to add the Project Root to path, run django.setup(), then run asgi.py from the Docker start scripts which is not what the Cookie Cutter setup does currently. Running Gunicorn with Uvicorn workers does solve this without any modifications, which makes me agree if Gunicorn had reloading work with Uvicorn workers, this might be a nice positive addition to making dev and prod as similar as possible. Perhaps this serves as an example of what happens when they aren't - things have to change.

Thanks, Omar

[Andrew-Chen-Wang]

@omarsumadi Taking a look at that traceback again, I don't have django.setup() in my asgi, wsgi, or manage.py. Take a look at cookiecutter Django's files again. I'll also take a look at that issue you rose.

[omarsumadi]

@omarsumadi Taking a look at that traceback again, I don't have django.setup() in my asgi, wsgi, or manage.py. Take a look at cookiecutter Django's files again. I'll also take a look at that issue you rose.

I don't either - just undid all the changes I suggested and ran it again (I have some comments in the file, so it doesn't match up exactly with Cookie Cutter's ASGI.py file). I should have probably told you I'm running Django Channels as well (but I know you don't like it haha)

django            | Process SpawnProcess-1:
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
django            |     self.run()
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
django            |     self._target(*self._args, **self._kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/subprocess.py", line 61, in subprocess_started
django            |     target(sockets=sockets)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 47, in run
django            |     loop.run_until_complete(self.serve(sockets=sockets))
django            |   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 54, in serve
django            |     config.load()
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 306, in load
django            |     self.loaded_app = import_from_string(self.app)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
django            |     module = importlib.import_module(module_str)
django            |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django            |     return _bootstrap._gcd_import(name[level:], package, level)
django            |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
django            |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
django            |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
django            |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
django            |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
django            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django            |   File "./config/asgi.py", line 29, in <module>
django            |     django_application = get_asgi_application()
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/asgi.py", line 12, in get_asgi_application
django            |     django.setup(set_prefix=False)
django            |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
django            |     apps.populate(settings.INSTALLED_APPS)
django            |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
django            |     app_config.ready()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/apps.py", line 23, in ready
django            |     ExportMigrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/migrations.py", line 52, in ExportMigrations
django            |     executor = MigrationExecutor(connections[alias])
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
django            |     self.loader = MigrationLoader(self.connection)
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 49, in __init__
django            |     self.build_graph()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 212, in build_graph
django            |     self.applied_migrations = recorder.applied_migrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 76, in applied_migrations
django            |     if self.has_table():
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 56, in has_table
django            |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
django            |   File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
django            |     raise SynchronousOnlyOperation(message)
django            | django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

My suggestion in the Issues you saw is a way to work around this issue I have here. Something is calling Django Setup from this package.

[Andrew-Chen-Wang]

@omarsumadi I see. I think it'll be wise to move this discussion to that issue then, since, in my belief, this issue should be closed as correct answers have already been given several times by multiple people, and your current problem is more Django related.

[alanwilter]

@Andrew-Chen-Wang And, what is the correct answer?

gunicorn main:app --reload -w 2 -k uvicorn.workers.UvicornWorker

still does not work for me. Simply using uvicorn is not the answer I'm looking for.

[Andrew-Chen-Wang]

@alanwilter Hm, I'm no guru on gunicorn, but try installing the inotify package. If that doesn't work, try using watchgod. It could just be gunicorn not picking up on sys notifs.

Edit: I think you need to add --reload-engine inotify

[sandys]

I face the same issue here on gunicorn

On Thu, 11 Mar, 2021, 20:53 Andrew Chen Wang, @.***> wrote:

@alanwilter https://github.com/alanwilter Hm, I'm no guru on gunicorn, but try installing the inotify package. If that doesn't work, try using watchgod. It could just be gunicorn not picking up on sys notifs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/benoitc/gunicorn/issues/2339#issuecomment-796815523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYU3UM7OXNTOIFZGLBGTTDDG5XANCNFSM4NKBSMLQ .

[sandys]

something very strange is happening with the latest version of gunicorn and uvicornworker.

This works ONE TIME gunicorn p:app -p 8080 --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker

gunicorn correctly picks up any file changes and does a hot-reload

unicorn p:app -p 8080 --reload --reload-engine inotify   -k uvicorn.workers.UvicornWorker                                                (git)-[master]-
[2021-03-17 14:35:42 +0530] [298736] [INFO] Starting gunicorn 20.0.4
[2021-03-17 14:35:42 +0530] [298736] [INFO] Listening at: http://127.0.0.1:8000 (298736)
[2021-03-17 14:35:42 +0530] [298736] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 14:35:42 +0530] [298739] [INFO] Booting worker with pid: 298739
[2021-03-17 14:35:43 +0530] [298739] [INFO] Started server process [298739]
[2021-03-17 14:35:43 +0530] [298739] [INFO] Waiting for application startup.
[2021-03-17 14:35:43 +0530] [298739] [INFO] Application startup complete.
[2021-03-17 14:35:48 +0530] [298739] [INFO] Worker reloading: p.py.a42bb679bd860346e8c7d564fdc6cc56.tmp modified

But after this one-reload, it stops working. It doesnt pick up any subsequent file changes. Not sure why its happening

[alanwilter]

I've seen it, but it actually is doing nothing. gunicorn main:app --reload -k uvicorn.workers.UvicornWorker

However, if I add inotify:

gunicorn main:app --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker
[2021-03-17 10:43:49 +0100] [31332] [INFO] Starting gunicorn 20.0.4
[2021-03-17 10:43:49 +0100] [31332] [INFO] Listening at: http://127.0.0.1:8000 (31332)
[2021-03-17 10:43:49 +0100] [31332] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 10:43:49 +0100] [31333] [INFO] Booting worker with pid: 31333
[2021-03-17 10:43:49 +0100] [31333] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/uvicorn/workers.py", line 63, in init_process
    super(UvicornWorker, self).init_process()
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 132, in init_process
    self.reloader = reloader_cls(extra_files=self.cfg.reload_extra_files,
TypeError: __init__() got an unexpected keyword argument 'extra_files'
[2021-03-17 10:43:49 +0100] [31333] [INFO] Worker exiting (pid: 31333)
[2021-03-17 10:43:49 +0100] [31332] [INFO] Shutting down: Master
[2021-03-17 10:43:49 +0100] [31332] [INFO] Reason: Worker failed to boot.

It doesn't work at all. Is there anything special to do about inotify? It seems a feature for Linux kernels, I'm running macOS Big Sur. Anyway I did pip install -U uvloop httptools inotify.

[alanwilter]

So I tested on Linux (Ubuntu 18.04) as well, failed. My whole setup (you need Python 3.7 or higher):

mkdir fastapi
cd fastapi

create file main.py

from typing import Optional
from fastapi import FastAPI
from pydantic import BaseModel

class Item(BaseModel):
    name: str
    description: Optional[str] = None
    price: float
    tax: Optional[float] = None

app = FastAPI()

@app.post("/items/")
async def create_item(item: Item):
    return item

Now, set python venv:

python3 -m venv venv
source venv/bin/activate
pip install -U fastapi gunicorn pip wheel uvicorn uvloop httptools

Standard way work as expected when changing main.py:

uvicorn main:app --reload
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [53638] using statreload
INFO:     Started server process [53640]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
WARNING:  StatReload detected file change in 'main.py'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [53640]
INFO:     Started server process [53758]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Now, try with basic gunicorn, inducing a change that would cause main.py to failed (like a typo returns item):

gunicorn main:app --reload -k uvicorn.workers.UvicornWorker
[2021-03-17 11:43:37 +0000] [95542] [INFO] Starting gunicorn 20.0.4
[2021-03-17 11:43:37 +0000] [95542] [INFO] Listening at: http://127.0.0.1:8000 (95542)
[2021-03-17 11:43:37 +0000] [95542] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 11:43:37 +0000] [95544] [INFO] Booting worker with pid: 95544
[2021-03-17 11:43:37 +0000] [95544] [INFO] Started server process [95544]
[2021-03-17 11:43:37 +0000] [95544] [INFO] Waiting for application startup.
[2021-03-17 11:43:37 +0000] [95544] [INFO] Application startup complete.
[2021-03-17 11:44:02 +0000] [95544] [INFO] Worker reloading: /home/awilter/fastapi/main.py modified

So, it may catch a "change", but it's not reloaded.

Using, inotify:

gunicorn main:app --reload --reload-engine=inotify -k uvicorn.workers.UvicornWorker
[2021-03-17 11:49:58 +0000] [99660] [INFO] Starting gunicorn 20.0.4
[2021-03-17 11:49:58 +0000] [99660] [INFO] Listening at: http://127.0.0.1:8000 (99660)
[2021-03-17 11:49:58 +0000] [99660] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 11:49:58 +0000] [99662] [INFO] Booting worker with pid: 99662
[2021-03-17 11:49:58 +0000] [99662] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line
 583, in spawn_worker
    worker.init_process()
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/uvicorn/workers.py", line
63, in init_process
    super(UvicornWorker, self).init_process()
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/gunicorn/workers/base.py",
 line 132, in init_process
    self.reloader = reloader_cls(extra_files=self.cfg.reload_extra_files,
TypeError: __init__() got an unexpected keyword argument 'extra_files'
[2021-03-17 11:49:58 +0000] [99662] [INFO] Worker exiting (pid: 99662)
[2021-03-17 11:49:58 +0000] [99660] [INFO] Shutting down: Master
[2021-03-17 11:49:58 +0000] [99660] [INFO] Reason: Worker failed to boot.

As for watchgod, I don't know how to use it, doing simply --reload-engine=watchgod does not work.

[c-py]

I think you get got an unexpected keyword argument 'extra_files' if gunicorn cannot detect you have inotify installed, either because you are not running on linux or there were import errors from inotify package. https://github.com/benoitc/gunicorn/blob/master/gunicorn/reloader.py#L121

In my case I'm running on Mac so it looks like I can't use the inotify engine and my only option is to use the poll engine. I don't know enough about gunicorn + uvicorn to know if the reloading has ever worked - but to me it looks like we need to get the uvicorn server to exit via SIGINT or SIGTERM.

In order to get the poll engine to reload the server I have added a small worker_int hook in my gunicorn.conf.py file which signals to exit the uvicorn server via SIGINT.

import os
import signal

def worker_int(worker):
    os.kill(worker.pid, signal.SIGINT)

This has allowed code reloading to work but it feels pretty barbaric. I'm sure there's a better way to achieve this. I run gunicorn like this: gunicorn main:app -w 4 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker --log-file=- --log-level DEBUG --reload

[sandys]

@c-py u seem to be on the right track. I'm roughly getting it to work on linux...however I'm having to save TWICE, not once to get this to work

[kylemacfarlane]

If you're already using supervisor inside a container then you can add a process like this to the end of your development config. You will need procps and inotify-tools installed for pkill and inotify-wait.

[program:inotifywait]
command=bash -c "inotifywait --excludei \"[^(\.py)]+$\" -e modify -r path-to-watch/ && pkill -SIGHUP gunicorn"
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

I like this method because it can be adapted to work with practically anything and you don't need to change anything else in production vs development.

[liquiddandruff]

something very strange is happening with the latest version of gunicorn and uvicornworker.

This works ONE TIME gunicorn p:app -p 8080 --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker

gunicorn correctly picks up any file changes and does a hot-reload

unicorn p:app -p 8080 --reload --reload-engine inotify   -k uvicorn.workers.UvicornWorker                                                (git)-[master]-
[2021-03-17 14:35:42 +0530] [298736] [INFO] Starting gunicorn 20.0.4
[2021-03-17 14:35:42 +0530] [298736] [INFO] Listening at: http://127.0.0.1:8000 (298736)
[2021-03-17 14:35:42 +0530] [298736] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 14:35:42 +0530] [298739] [INFO] Booting worker with pid: 298739
[2021-03-17 14:35:43 +0530] [298739] [INFO] Started server process [298739]
[2021-03-17 14:35:43 +0530] [298739] [INFO] Waiting for application startup.
[2021-03-17 14:35:43 +0530] [298739] [INFO] Application startup complete.
[2021-03-17 14:35:48 +0530] [298739] [INFO] Worker reloading: p.py.a42bb679bd860346e8c7d564fdc6cc56.tmp modified

But after this one-reload, it stops working. It doesnt pick up any subsequent file changes. Not sure why its happening

Experiencing the same exact behaviour here on Mac, the reload works only once. I didn't specify --reload-engine though.

After adding the gunicorn.conf.py by @c-py it works well enough for now. Thanks.

[jvasi]

The issue seems to be in the gunicorn -> workers/base.py -> init_process() -> def changed(fname) method. This function is called by the change watcher thread and it ends with sys.exit(0) which, unfortunately, results in only killing that thread (that's why changes are detected only once).

To workaround the issue, you can define a custom UvicornWorker with additional thread that sends a KILL signal to the current process if the worker's active flag is set to False (which is done by the changed(fname) function above).

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker

class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)

class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

To use that worker you need to specify it with the gunicorn's worker_class parameter, e.g.:

gunicorn -k some_package.RestartableUvicornWorker ...

[rotten]

Thanks @jvasi that worked great.

I get one INFO notification whenever gunicorn goes down this way:

[INFO] Error while closing socket [Errno 9] Bad file descriptor

I believe this is coming from this line in gunicorn: https://github.com/benoitc/gunicorn/blob/1299ea9e967a61ae2edebe191082fd169b864c64/gunicorn/sock.py#L69

[fvalverd]

This seems related to https://github.com/benoitc/gunicorn/issues/2604

[abbazs]

The issue seems to be in the gunicorn -> workers/base.py -> init_process() -> def changed(fname) method. This function is called by the change watcher thread and it ends with sys.exit(0) which, unfortunately, results in only killing that thread (that's why changes are detected only once).

To workaround the issue, you can define a custom UvicornWorker with additional thread that sends a KILL signal to the current process if the worker's active flag is set to False (which is done by the changed(fname) function above).

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker

class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)

class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

To use that worker you need to specify it with the gunicorn's worker_class parameter, e.g.:

gunicorn -k some_package.RestartableUvicornWorker ...

This solution works. Setting --reload-engine=inotify alone does not work.

[rykroon-momentum]

The issue seems to be in the gunicorn -> workers/base.py -> init_process() -> def changed(fname) method. This function is called by the change watcher thread and it ends with sys.exit(0) which, unfortunately, results in only killing that thread (that's why changes are detected only once).

To workaround the issue, you can define a custom UvicornWorker with additional thread that sends a KILL signal to the current process if the worker's active flag is set to False (which is done by the changed(fname) function above).

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker

class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)

class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

To use that worker you need to specify it with the gunicorn's worker_class parameter, e.g.:

gunicorn -k some_package.RestartableUvicornWorker ...

Should this be requested as a change to Uvicorn?

Often times when I am writing an async WebAPI in python, regardless of the framework I am using, I always come across this issue and find myself googling the problem just to come back to this specific thread and I end up copying this exact code every time.

Curious as to what others think?

[Soykertje]

If someone faces the following error using gunicorn+uvicorn workers: ModuleNotFoundError: No module named 'uvloop', this helped me to make it work:

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker

class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)

class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "asyncio", "http": "h11"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

It's basically the same code just changing the loop to asyncio and http to h11. To use it is exactly the same as pointed by @jvasi:

gunicorn -k some_package.RestartableUvicornWorker ...

[benoitc]

no activity since awhile. closing feel free to create a new ticket if needed.