benoitc / gunicorn

gunicorn 'Green Unicorn' is a WSGI HTTP Server for UNIX, fast clients and sleepy applications.
http://www.gunicorn.org
Other
9.76k stars 1.75k forks source link

Gunicorn is not restarting workers on --reload-extra-file change #2445

Open jkugler opened 3 years ago

jkugler commented 3 years ago

[I could not figure out how to post to the mailing list or forum, so am creating this issue. Additionally, it appears I cannot add labels when I create the issue.

I am starting gunicorn in a docker container like so:

ENTRYPOINT ["/usr/bin/gunicorn3", "--bind", ":8080", "--workers", "4", "--chdir", "/app", "main:app", "--reload-extra-file", "/mount_point/config/config.json", "--reload-engine", "inotify"]

This is gunicorn 20.0.4.

I have tried both inotify and poll for the reload engine.

/mount_point/config/config.json is a docker volume mounted at invocation.

The file does exist, because my web server inside the container will serve it. When I change the file outside the container, the web app serves the update file, BUT gunicorn gives no indication it has killed/restarted any workers and creating an end point (ex. /config_at_start) which serves a copy of the file loaded at start time (and not each time the end point is hit) keeps showing the old file, indicating the worker has not been restarted.

Any pointers in debugging this issue?

jkugler commented 3 years ago

I did further debugging, outputting the result of os.stat from my app. The file I'm watching does have its st_mtime (modification time) change when I modify it from outside the container, so Docker is updating the file metadata.

Again, I tried both "inotify" and "poll" engines.

jkugler commented 3 years ago

I've done some more digging. I found this code (open source for the win!):

        # start the reloader
        if self.cfg.reload:
            def changed(fname):
                self.log.info("Worker reloading: %s modified", fname)
                self.alive = False
                os.write(self.PIPE[1], b"1")
                self.cfg.worker_int(self)
                time.sleep(0.1)
                sys.exit(0)

            reloader_cls = reloader_engines[self.cfg.reload_engine]
            self.reloader = reloader_cls(extra_files=self.cfg.reload_extra_files,
                                         callback=changed)

It would appear that 1) You must pass --reload if you want --reload-extra-file to work, and 2) this isn't documented. In my case, I did not want (nor need) my code files to be monitored, as they are copied in to the container on build, and never change for the life of the container. I only wanted the config file to be monitored.

So, I guess this is a doc update bug, in the end.

Side note: I don't see in the anywhere in the code where the main file (the wsgi_app passed on command line, e.g. main:app) or any other code is added to the reloader. Did I miss that? If I didn't miss anything, maybe the documentation should be clarified to "--reload enables the reloader, and --reload-extra-file specifies files which will be monitored. No files are monitored by default.

cs-alok-agarwal commented 1 year ago

We also looking for an option where we do not want to monitor the entire code base but only specific file set needs to be monitored for reloading.

We observed that ading --reload along with --reload-extra-file it monitor entire code plus additionally the file you have specified.

Having option like --touch-reload=path/to/file like is uwsgi will be of great add.